Blog

Rich in the Jungle: A AWS to Softlayer comparison for PostgreSQL

I have updated my Rich in the Jungle presentation with new pricing for AWS vs. Softlayer. Things haven't changed much, in terms of raw performance per dollar (which is not the only qualifier). Softlayer is clearly the winner.

Read More

The fall of Open Source

Once upon a time FOSS was about Freedom. It was about exposing equality within source code. It allowed everyone equal rights and equal access to the technology they were using. An idea that if you were capable, you could fix code or pay someone to fix code. An ideology that there was something greater than yourself and that there was an inherent right built into what it is to be human with software.

Leaders to lemmings

I sat in a bar slowly nursing beers with other community members over a period of hours. We spoke of many things. We spoke of the never-done new PostgreSQL website. We spoke of my distaste for Amazon Web Services since reformed, with the exception ...

Read More

What is good for the community is good for the company (profit is the reward)

As the PostgreSQL community continues down its path of world domination I can't help but wonder whether the various PostgreSQL companies are going to survive the changes. Once upon a time there was an undercurrent of understanding that what was good for the community was good for the company. Whatever company that may be. However, over the last few years it seems that has changed. It seems there is more prevalance toward: What is good for the company is good for the community, or in other words, "The goal is profit."

That is a flawed discipline to follow in the Open Source world. A truly beneficial, strong and diverse community has to eliminate that thought entirely. The goal is ...

Read More

PgConf.US: 2016 Kicking the donkey of PostgreSQL Replication

My slides from my presentation and PgConf.US 2016:

 

Read More

Spreading the conference love

The PostgreSQL community has a lot of conferences in the United States:
  • PgUS United States PostgreSQL Conference 
  • Citus Data PgConfSV
  • PgUS SCALE PgDay (which as of 2016 is really a conference within a conference)
  • PostgresOpen
  • EDB PostgresVision
And that doesn't come even close to the number of various conferences in Europe.

As Bruce Momjian pointed out in his excellent blog this is a good thing. It is true that in the United States there is the big boy on the block and it will likely hit 600 people in 2017 but other than that the conferences are all small and regional. The exception is PostgresVision which hasn't yet run and therefore we don't know how many people ...

Read More

You are my fellow community member

I attended the fantastically presented PgConf US 2016 last week. An amazing conference, my training was well attended, my talk was at capacity, the 20th Anniversary Party was phenomenal and the conference raised money for an excellent cause. There were over 435 attendees, giving our brothers and sisters at PgConf EU something to work for during their conference in November.

While attending the hallway track, I was talking to a gentleman whose name escapes me. He asked me how he could contribute to the community. I am always excited to have that conversation because we are able to discuss all kinds of different ways to contribute, whether it be social (user groups, pgday, speaking at alternative conferences), documentation, code, or ...

Read More

PostgreSQL, FOSS, SCALE, NYCPUG, SPI, LFNW and Ruby oh my!

Three weeks ago I was in Pasadena for SCALE 14. I had over 100 people in my room as I blistered the behind of PostgreSQL and how it handles backups. If you are interested in seeing my considered opinion I will also be training on the same topic at PgConf.US. I am also speaking on PostgreSQL Replication and finally, I was told that I am running the Lightning Talks. I have never coordinated something like Lightning Talks before. It should be an interesting experience.

If that wasn't enough news, I have more! The Ruby community has adopted the draft PostgreSQL Code of Conduct. As one of the primary authors of that document, I am honored to have such ...

Read More

.Org developer meeting @ FOSDEM

A lot of people probably don't know this but PostgreSQL does plan. It is true that we take all contributions and they are reviewed based on their merit but it is also true that the community tries very hard to have a road map of some sort. Those road maps are created by the more prolific contributors in the community.

In the past there was a yearly Developer Meeting. That meeting would take place at PgCon. PgCon is held in May at the University of Ottawa, Canada. It is a small but great developer conference.

This year we are going to have two plus probably an informal one for a total of three. The first of which is taking ...

Read More

Scale 14x, PostgreSQL mini-conf, PgConf.US and NYCPUG

It is really not fair to call it a mini-conf. The Scale 14x, PostgreSQL Day attendance was larger than every conference except PgConf.US (EDIT: in the United States/Canada). It is a great opportunity to integrate with a wider community that is diverse, technologically capable and at the front lines of production installations.

I spoke on Backups: The Good, the Bad, and the Ugly. I had over 100 attendees in my room. It was obvious throughout the talks that this is going to be the new West coast conference for PostgreSQL. The opportunity for advocacy and integration into alternative technologies is just too great to ignore from either PostgreSQL or the wider FOSS community that attends SCALE.

This is ...

Read More

13:58, the sun is shining, the sky is blue, and I just had the best tomatoes -- ever. Welcome to Vienna and PgConf.eu

I am sitting in a glorious (although not the conference) hotel, writing this article. It is 13:58, at least it is where I am from, the great Pacific Northwest. I must admit, I miss the trees although I hear there are lots of them if you leave the city. What is there to say about this great city that the European community picked for their latest conference?

First and probably obvious, it is full of history. Second, don't bother doing anything on a Sunday. Third, smoking is allowed in restaurants (one ridiculous thing I have noted). Fourth, they sell water that has, "Natural Oxygen". Fifth, it feels a lot like Paris but the people seem friendlier. Lastly, this ...

Read More

PostgreSQL 9.5, Community, Features and More!

I spoke at the Whatcom PUG meeting last night on PostgreSQL 9.5. This is my fourth time giving this talk. The previous locations were DCPUG, PhillyPUG, and NYCPUG last month. The talk was well received and this was Whatcom PUG's best turn out yet! We even had an Open Street Map developer visit from Vancouver B.C.

The presentation does discuss some of the more popular features of 9.5, but as a whole it discusses the state of PostgreSQL as of 9.5. That includes features, community, and process. I think the most important item is the user interaction. At each presentation location I brought up the fact that PostgreSQL has no bug/issue tracker. This led ...

Read More

Tip for West side U.S. folks going to PgConf.EU in October

This tip works very well for me because of my physical location (Bellingham, WA) but it would also work reasonably well for anyone flying from Denver->West Coast including places such as Houston. It does take a little bit of patience though.

A normal trip for myself would mean driving down to SEA which is 90 minutes to 2 hours. This year, I decided on whim to see what it would take to fly out of YVR (Vancouver, B.C.) which is only 60 minutes driving.

Since I would be flying out of YVR on a non-connecting flight, I paid Canadian Dollars. For those that haven't been paying attention, the U.S. dollar has been doing very well lately ...

Read More

Elevating your confidence with the Elephant's restoration capabilities

In the beginning

There was Unix, Linux and Windows. They all run on hardware and that hardware all has bugs. What is the best way to work around hardware bugs? Backups.

You haven't had bad hardware, only bad developers? That's o.k., we have a solution to them too. It is called backups.

You haven't had bad hardware or bad developers, just bosses who still demand to have direct access to the data even though they haven't proven an ability to extract useful information without an extreme amount of hand holding? That's o.k., we have a solution to them too. It is called backups.

You haven't had any of the above? Lucky you ...

Read More

A new user discovers the PostgreSQL public schema

A new user of PostgreSQL recently discovered that PostgreSQL allows any PostgreSQL user to create objects and data by default.[1] I know you are saying, "What... PostgreSQL has some of the most advanced and flexible security in the industry!" and you are absolutely correct, we do. However, once you can connect to PostgreSQL, you have some interesting default capabilities. Consider the following example:

postgres@sqitch:/# psql -U postgres
psql (9.2.11)
Type "help" for help.

postgres=# create user foo;
CREATE ROLE
postgres=# \q

No biggy, we created a user foo as the super user postgres. All is good. However, what can that user foo do?

postgres@sqitch:/# psql -U foo postgres
psql (9.2.11)
Type "help" for ...

Read More

Let's delete contrib!

There has been a lot of discussion about the upcoming extension pg_audit and whether or not it should be in contrib. You can read about that here. The end result of the discussion is that pg_audit is going to be reverted and not in contrib. There were plenty of technical reasons why people didn't want it in contrib but I have a different reason. It is an extension. It doesn't need to be in contrib. In fact, I argue that because of pgxs and extensions we don't need contrib at all. If you don't follow the mailing lists my argument is blow and please feel free to comment here. The discourse is very much needed on ...

Read More

Updating the .Org docs on backups

I spent a great deal of time working through the SQL DUMP portion of the 9.5devel docs this past week. Below is the current text of what I have and it would be great if my readers would take a look and offer some thoughtful feedback. What would you like to see added? What would you like to see changed? Please note that this is reference documentation not tutorial documentation.

This is just the straight HTML dump that is generated from Docbook but since it is inline the links won't work. The current -devel docs are here and the updated version I am working is below:


24.1. SQL Dump

PostgreSQL provides the program pg_dump for generating a ...

Read More

WhatcomPUG meeting on 04/21. Start date, end date, calculate

The PUG meeting was good. We now have a consistent if small group that are attending. Before the presentation we spoke about possibly moving the group to meetup to get a little better visibility. G+ Communities are awesome but Meetup seems to be where the people in the area look.

The presentation was provided by Eric Worden who happens to be a CMD employee. The talk overall is very good and provided a lot of information that I didn't know about dates. It also lead to the development of a new PostgreSQL extension (more on that at a later time).

The most interesting part of the talk to me was the use of a dimensions table to make date ...

Read More

Reflections on PgConf.US 2015

Saturday the 18th of April, I woke up to the following:
 

It was one of those moments that you realize just how blessed of a life you have. A moment where you stop and realize that you must have done something right, at least once. I was with my all of my ladies, there were no other people at the camp site, the weather was clear and it was set to hit 68F. The only sound was the gentle lapping of water and the occasional goose.

It was at this time that I was able to finally take a step back and reflect on PgConf.US. This conference meant a lot to me professionally. I didn't organize it ...

Read More

WhatcomPUG meeting last night on: sqitch and... bitcoin friends were made!

Last night I attended the second WhatcomPUG. This meeting was about Sqitch, a interesting database revision control mechanism. The system is written in Perl and was developed by David Wheeler of PgTap fame. It looks and feels like git. As it is written in Perl it definitely has too many options. That said, what we were shown works, works well and appears to be a solid and thorough system for the job.

I also met a couple of people from CoinBeyond. They are a point-of-sale software vendor that specializes in letting "regular" people (read: not I or likely the people reading this blog) use Bitcoin!

That's right folks, the hottest young currency in the market today is using the ...

Read More

Stomping to PgConf.US: Webscale is Dead; PostgreSQL is King! A challenge, do you accept?

I submitted to PgConf.US. I submitted talks from my general pool. All of them have been recently updated. They are also all solid talks that have been well received in the past. I thought I would end up giving my, "Practical PostgreSQL Performance: AWS Edition" talk. It is a good talk, is relevant to today and the community knows of my elevated opinion of using AWS with PostgreSQL (there are many times it works just great, until it doesn't and then you may be stuck).

I also submitted a talk entitled: "Suck it! Webscale is Dead; PostgreSQL is King!". This talk was submitted as a joke. I never expected it to be accepted, it hadn't been written ...

Read More

PostgreSQL is King! Last week was quite busy being a servant.

Last week was one of the busiest community weeks I have had in a long time. It started with an excellent time in Vancouver, B.C. giving my presentation, "An evening with PostgreSQL!" at VanLUG. These are a great group of people. They took all my jibes with good humor (Canadians gave us Maple Syrup, we gave them Fox News) and we enjoyed not only technical discussion but discussions on technology in general. It is still amazing to me how many people don't realize that Linux 3.2 - 3.8 is a dead end for random IO performance.

After VanLUG I spent the next morning at the Vancouver Aquarium with my ladies. Nothing like beautiful weather, dolphins and jelly ...

Read More

AWS performance: Results included

I am not a big fan of AWS. It is a closed platform. It is designed to be the Apple of the Cloud to the Eve of Postgres users. That said, customers drive business and some of our customers use AWS, even if begrudgingly. Because of these factors we are getting very good at getting PostgreSQL to perform on AWS/EBS, albeit with some disclosures:
  1. That high IO latency is an acceptable business requirement.
  2. That you are willing to spend a lot of money to get performance you can get for less money using bare metal: rented or not. Note: This is a cloud issue not an AWS issue.

Using the following base configuration (see adjustments for each configuration after ...

Read More

Don't kill yourself

As a PostgreSQL consultant you end up working with a lot of different types of clients and these clients tend to all have different requirements. One client may need high-availability, while another needs a DBA, while yet another is in desperate need of being hit with a clue stick and while it is true that there can be difficult clients, there is no bad client.

What!!! Surely you can't be serious?

Don't call me shirley.

I am absolutely serious.

A bad client is only a reflection of a consultants inability to manage that client. It is true that there are difficult clients. They set unrealistic expectations, try to low ball you by with things like: "We can get ...

Read More

Along the lines of GCE, here are some prices

I was doing some research for a customer who wanted to know where the real value to performance is. Here are some pricing structures between GCE, AWS and Softlayer. For comparison Softlayer is bare metal versus virtual.

GCE: 670.00
16 CPUS
60G Memory
2500GB HD space

GCE: 763.08
16 CPUS
104G Memory
2500GB HD space

Amazon: 911.88
16 CPUS
30G Memory
3000GB HD Space

Amazon: 1534.00
r3.4xlarge
16 CPUS
122.0 Memory
SSD 1 x 320
3000GB HD Space

Amazon: 1679.00
c3.8xlarge
32 CPUS
60.0 Memory
SSD 2 x 320
3000GB HD Space

None of the above include egress bandwidth charges. Ingress is free.

Softlayer: ~815 (with 72GB memory ~ 950)
16 Cores ...

Read More

GCE, A little advertised cloud service that is perfect for PostgreSQL

Maybe...

I have yet to run PostgreSQL on GCE in production. I am still testing it but I have learned the following:

  1. A standard provision disk for GCE will give you ~ 80MB/s random write.
  2. A standard SSD provisioned disk for GCE will give you ~ 240MB/s.

Either disk can be provisioned as a raw device allowing you to use Linux Software Raid to build a RAID 10 which even further increases speed and reliability. Think about that, 4 SSD provisioned disks in a RAID 10...

The downside I see outside of the general arguments against cloud services (shared tenancy, all your data in a big brother, lack of control over your resources, general distaste for $vendor, or whatever else ...

Read More