Posted Wednesday Jul 28th, 2010 09:35am
by Joshua Drake |
Permalink
I expected feedback from the community on
FOSSExperts. I did not expect feedback with such immediacy. All the feedback I have received so far is positive. Which is a great feeling. Here are the key points that are coming back.
How do deal with disagreement about the deliverable:
This is an interesting one. I wanted to keep FOSSExperts simple. That is why the deliverable on the ALTER TABLE project is simple, committed to PostgreSQL Core.
That may not work in all circumstances. So I am considering one of two options. The first option is courtesy of Josh Berkus. The idea would be to have a board of people that determine whether or not the deliverable has been met. This has merit because you have a panel of experts that make the determination. It can make the review process painless but is also takes the power out of both the funders and developers.
The second one is to take a vote. It would work something like this.
- Developer states project is complete and demonstrates completeness based on the deliverable.
- Every person who funded the project votes on whether or not the project is complete.
- If 66% vote the project complete, developer gets their money.
- If less than 66% vote project incomplete, developer doesn't get their money.
There would have to be some caveats. First the funders need to be able to communicate with the developer because they may not have understood part of the spec. Further as we are working with Open Source, the end deliverable may have been changed based on the will of the community versus the developer (see the Hot Standby work with PostgreSQL).
The voting would also need to be limited to a period of time. I was thinking 14 days. The 66% would be tallied against those that voted in that 14 days.
I like this idea because it removes the third party and it stops a single funder from calling foul as they are part of a collective vote. What do you think?
Other than that the feedback has been extremely positive. I even posted to the LedgerSMB list and multiple people are excited to see this opportunity.
If you have ideas,
please share them. We have setup a flame page just for this.
Remember a lot of your questions can be answered in the FAQ as well.
Posted Tuesday Jul 27th, 2010 03:37pm
by Joshua Drake |
Permalink
The cat is out of the proverbial bag. I originally planned to have a quiet roll out with a few close contributors but that has gone by the wayside. Now I am going to be pushing hard for people to test, beat on, object to, argue about, flame upon, scream at, praise and hopefully help us build out something that is truly useful for the FOSS Community. What am I blathering on about?
FOSSExperts of course.
FOSSExperts is a new site specifically engineered to allow FOSS developers to raise money for projects they are trying to develop. The idea stemmed from the very cool Kickstarter. With our focus obviously being on a different kind of creative.
This is long overdue in the FOSS Community. There are a great deal of communities out there (LedgerSMB for example) that can use a place for their developers to try and raise funds for a specific feature. LedgerSMB just recently had a discussion on developing a Payroll module. Developing a Payroll module will be expensive for a single small company to absorb, but 20 small companies? Not nearly as expensive.
What FOSSExperts is not, is a place to send money to global projects such as Debian or PostgreSQL.org. It is for specific, well defined proposals and has a specific and defined delivery as well as refund policies etc.
Right now, we are in closed Beta. If you have a project or proposal you would like to try out you need to email me directly but we are interested. So take a look, and let the rage begin! If you like, you can review one of the larger proposals already on the site as well.
Posted Tuesday Jul 20th, 2010 01:13pm
by Aurynn Shaw |
Permalink
Following up on the blog post covering the new coolness in 0.3, and better docs on working with Simpycity, we've just released Simpycity 0.3.1, our best release yet!
Simpycity can be
downloaded from our Wiki, and our code is available from the
Subversion repository.
Finally, starting today, all new releases of Simpycity are available on the
PyPI package index, and Simpycity installable via:
$ easy_install Simpycity
Posted Tuesday Jul 20th, 2010 01:04pm
by Aurynn Shaw |
Permalink
Simpycity is, as
we've previously covered, a small library that permits for the direct mapping of arbitrary SQL statements to Python callables.
This power allows for the development of complex representations that do not need to directly map to the underlying database representation.
This differs from most conventional ORM technology, which follows the ActiveRecord pattern.
Simpycity was implemented in this way for a great many reasons, first and foremost that the Active Record pattern is not the best representation of your data to your application logic. Should the application need to be aware of underlying data relationships? Should
the application be aware of foreign keys, structures, and other underlying constructs?
Or should the application be able to interact with the data in a form that is logical, and sensible to the application, without needing deep
knowledge of underlying representations?
We thought so, and Simpycity, and a concept more along the lines of Active Object is our result.
Disparate Representations
Simpycity, instead of writing SQL for you via query generators, requires that the developer write SQL by hand.
The reason for this is that database representations are not generalizable into object relations - this disconnect is the entire
reason behind the object-relational difficulties.
A proper object representation encapsulates all the possible information about a method in a single location, as well as all the necessary
methods to act on that data. A single object then represents a single quantum of data.
However, for a relational system, normal form requires that disparate pieces of information are further broken down, into points of absolute
truth about the data. A person's name, for instance, is a point of absolute truth, and should exist in only a single place in the database,
whereas a person's name could exist in several places in an Object system, in a sensible manner.
The disparity comes in that a Person, in terms of business requirements, is rather different from a Person in SQL terms, to the point where it would not be sensible to represent a database Person as an object Person - Address information, birthdate, all sorts of ancillary data that would normally be present isn't, per correct normal form.
Simpycity works to avoid this, by allowing for business models that have little if anything in common with the underlying table structure,
allowing for proper normalization as well as useful business objects.
Forging Anew
As Simpycity does not impose the database structure on your objects, it can't immediately provide the functionalities of .new() in the way a conventional ORM can - even though we've seen Simpycity handle the .save() feature brilliantly.
Instead, if you instance a Simpycity model, not from the database, you get precisely and only a Simpycity instance. As it's not connected to a known set of database data, all the functions and other associated items have no way of operating, and the model just sits there, forlorn and empty.
But since we have to match the Active Record pattern, how would we go about providing .new() in Simpycity?
Here's how we do it:
Given a standard model that looks like this,
class model(ctx.Model()):
table = ['id','name']
__load__ = ctx.Function("load_obj", ['id'])
We're able to do simple and basic load operations. Right?
But, to create a new object, the pattern more resembles:
class model(ctx.Model()):
...
new = ctx.Function("new_obj",['name'], return_type=model)
Which allows for the external interface of:
import yourmodel
o = yourmodel.new("Some name")
o.commit()
Providing a clean and sensible model API, following the ActiveRecord pattern,
but still offering all the power of Simpycity.
Twisty Little Properties
Another very nifty capability of ActiveRecord systems is that of reflection, automatically retrieving the far end of a foreign key constraint. This allows for useful functionality like
aModel.comments
correctly reaching across the one-to-many relationship and pulling all the comments.
As Simpycity doesn't directly map tables, capabilities such as this aren't directly implemented in Simpycity.
However, since we do realize that business objects need to perform similar tricks and load data in via properties, we added specific support for this into Simpycity.
But, since Simpycity is entirely callable based, we had to be able to support this feature with our existing metaphors. To that end, we included a simple function that will take any Simpycity callable (or any callable, really), interrogate its argument list, and handle argument mapping as you'd expect.
Using this feature is as simple as:
from simpycity.helpers import prop
class myTextObject( ctx.Model() ):
table = ['id', 'value']
__load__ = ctx.Function("textobject.by_id",['id'])
comments = prop( get=ctx.Function("textobject.comments", ['id']) )
mto = myText(1)
comments = mto.comments
Easily allowing for sensible properties to be created, based entirely on clean Simpycity code. Properties created in this way even support set and delete functionality, identical to a standard property, allowing for property accessors to easily manipulate the database layer.
As a note, prop() is a new feature in 0.3.1. 0.3.0 and below should use
from simpycity.helpers import sprop
class myTextObject( ctx.Model() ):
...
comments = property(sprop( get=ctx.Function("textobject.comments", ['id']) ))
Obviously not as clean, and not as capable. You should upgrade ASAP.
Next
We've stabilized the API, made everything work through the consistent Context interface, and have built a powerful callable-based model infrastructure for all sorts of application development.
So what's next for Simpycity? Well, some of the things we're planning on include breaking the Model object away from
psycopg2 dependency, allowing us to use other PG drivers (such as pg8000), as well as opening up the Model protocol we've defined for other contexts - file access, for instance. Anywhere that an application needs to represent a complex underlying structure as a simple object, the Model could be used.
More in the future, we're really looking forward to integrating Simpycity callables with Django and SQLAlchemy model objects, using Simpycity to provide strong, clean functional and raw query support in those environments. And vice-versa as well: Binding a SQLAlchemy or Django ORM chain to a Simpycity object, using it to populate an object, and building even more complex, effective business objects for your application.
Even farther afield, we've been looking at integrating query generation to Simpycity. There's a lot of boilerplate SQL that needs writing, and being able to hand it to an elegant, PostgreSQL-focussed abstraction would be, we think, ideal.
As always, Simpycity is
available on our Wiki, and our code can always be checked out from
our Subversion repository.
Finally, Simpycity is easily installed from the
PyPI index, using easy_install Simpycity.
Posted Saturday Jul 10th, 2010 12:22pm
by Joshua Drake |
Permalink
I am not writing this to jump all over
Big Jim's post but after reading it and seeing the syntax of Scala (and Java), I can't help but wonder, why anyone would use either language (based on syntax). Yes I know it is a matter of taste and everyone has an opinion. Let's just say my taste lean toward more succinct code.
#!/usr/bin/python
#
# Set up initial work
#
import psycopg2
conn = psycopg2.connect("dbname='postgres' user='postgres'")
cur = conn.cursor()
cur.execute("SELECT * FROM pg_database")
def output(cur):
#Run two queries, one for headers, one for data
tuples = cur.fetchall()
colname = [x[0] for x in cur.description]
buff = "\t" + "\t".join(colname[0:4]) + "\n"
for row in tuples:
# This is a little one liner but could easily be expanded
# for readability
buff += "\t" + "\t".join([str(i) for i in row[0:4]]) + "\n"
return buff
print output(cur)
I keep looking back at Java merged/derived/munged/glued languages,
Groovy looks interesting and of course there is
Jython but I think I will stick with good old fashion CPython just as I am sure that
MST will stick with Perl.
Posted Thursday Jul 8th, 2010 11:06am
by Joshua Drake |
Permalink
If you are running any version of PostgreSQL 7.4, 8.0 or 8.1, it is now time to upgrade to 8.3 or 8.4. The versions 7.4 and 8.0 are slated for end of life at the end of this month. The 8.1 version is slated for end of life in November.
This is not an item to take lightly. Once a version is end of life you will not be able to get support (easily), there will be no more security updates and no bug fixes even if they are data loss bugs.
I often find it disturbing how many people will run older releases. I am not talking about someone running 8.2 when 8.4 is out but we still see the occasional post on the lists about someone running 7.3!
Remember folks, at a minimum keep your dot releases updated. The community does not release dot releases on a whim, it is for the protection of your data.
Of course, if you need any help with
upgrading don't hesitate to ask.
Posted Tuesday May 11th, 2010 08:07pm
by Aurynn Shaw |
Permalink
It's been a long time since we shipped the first version of Simpycity, a long time since we've really discussed how it works and how to get the most of it.
Over the next couple of articles, I'm going to be discussing how we're using Simpycity internally, some ideas that we have going forward, and how you too can benefit.
Since we're just shipping Simpycity 0.3 now, we should go over some of the excellent new features available now.
Contexts
A bug we kept running into in Simpycity was related to Python's object lifecycle. Time and time again, our connections weren't being closed properly, held on to for far longer than they were useful.
We tried to fix this in 0.2 with the Manager, but that wasn't successful. While excellent in theory, a Manager would only run at the end of a given transaction, while a particular loop that spawned a lot of Simpycity objects could still easily exhaust our connection pool.
To combat this, we have implemented the Context. A Context is the object from which all modern Simpycity functionality derives, and it is used like this:
from simpycity.context import Context
ctx = Context(dsn="database=%s user=%s password=%s" % (database, username,
password))
Now, any standard Simpycity object can be constructed through the Context, and a single Context will keep all objects spawned from it under a single database connection. No more weirdness with resource exhaustion, and no more using the admittedly inconsistent simpycity.config object system.
Basic Queries
Now that we have a single, unified Context, spawning our basic primitives is just as easy, simply:
my_sproc = ctx.Function("my_getter",['id'])
or, for raw SQL:
my_raw = ctx.Raw("SELECT * FROM my_table")
Just as easy as Simpycity has ever been.
Models
Spawning models is, again, just as easy in Simpycity 0.3 as it has been in previous versions, though we now have some truly useful knowledge on how to use a Simpycity model effectively and with even greater ease than before.
For starters, the ability to get a column value from a model was limited at best. There was no default mechanism to load the value on a simple
model.column request.
As of 0.3.1, this has changed.
model.column is now supported by default on all Simpycity models.
Not only that, but we now support setting columns in the same fashion:
model.column = "a new value"
But, it doesn't normally propagate to the database. We're only manipulating an object within Python itself, not the underlying schema.
For that, we require a save mechanism.
Classically, Simpycity models assumed that manipulating the underlying database would occur via procedures, .Raw or .Function methods bound to the object.
While this is still an excellent metaphor, it does incur a penalty of several modifications each requiring a round-trip to the database; hardly an efficient mechanism.
A Save Mechanism
To add a more efficient mechanism to Simpycity, models now, by default, offer the .save() mechanism. This functionality works in two simple, easy parts.
The first requires that the model have a new bound method, specifically:
class myModel(ctx.Model()):
table = ['id','value']
__save__ = ctx.Function("update_table",['id','value'])
Then, on a model, one may:
>>> m = myModel(1)
>>> m.value
'a Value'
>>> m.value = 'New Value'
>>> m.value
'New Value'
>>> m.save()
>>> ctx.commit()
Thusly updating a simple model, to the database, easily and compactly.
By defining __save__ on a Simpycity model, it is simple and easy to succinctly save data in a consistent fashion.
For more complex save mechanisms, the standard Simpycity bind functionality remains, allowing for a model to host arbitrary functions that are able to read the underlying columns, as so:
class model(ctx.Model()):
table = ['id','value']
comments = ctx.Raw("SELECT * FROM comments WHERE table_id = %s",['id'])
Which is then called via:
m = model(1)
comments = m.comments()
Loading from the Database
As demonstrated above, Simpycity's models are able to save easily and quickly, in a fully customized fashion.
But what of loading data, an equally crucial part of the Model interface?
In this instance, Simpycity offers several mechanisms to allow for easy loading of data from your database.
First, each Model allows for a method to be run when the model is instanced, IF the model is instanced with an argument.
Therefore, a model instanced as:
m = model()
would create an empty object from the base class, and would in general be unable to save itself to the database with any ease.
However, if we were to do this:
class model(ctx.Model()):
table = ['id','value']
comments = ctx.Raw("SELECT * FROM comments WHERE table_id = %s",['id'])
__load__ = ctx.Raw("SELECT * FROM my_table WHERE id = %s",['id'])
m = model(1)
then Simpycity will execute __load__ during the model instance, loading the record from the database as expected.
This model is now able to be modified as described above, via the normal save functionality.
The second method that Simpycity provides for loading data is via the standard primitives, Raw and Function.
By providing a "model=" argument (previously "return_type"), returned rows from the query will be mapped into the provided model object.
This functionality can be used in multiple ways; first, to add functions to a model that return other models, such as:
class Comment(ctx.Model()):
table = ['id','owner','user']
class table(ctx.Model()):
table = ['id','value']
comments = ctx.Raw("SELECT * FROM comments WHERE table_id = ?",
['id'], model=Comment)
And thus, by doing:
m = table(1)
cmts = m.comments()
for comment in cmts:
# Something interesting with each comment object.
allowing for many-to-many relationships to be easily and elegantly expressed.
Furthermore, the model= argument to a primitive can be used to implement alterative loading mechanisms, bypassing the general __load__ method.
By performing:
class table(ctx.Model())
table = ['id','value']
by_id = ctx.Function("table.by_id",['id']. model=table)
by_value = ctx.Function("table.by_value",['value'],model=table)
it is easy to declare alternative instancing mechanisms, that fully match your business model requirements.
Next, we'll be covering to the best practises for full model packages, providing additional loading methods cleanly and easily - a structure we've taken to calling Active Object, as opposed to Active Record.
And, as always, you can get Simpycity from
the Wiki or
the repository.
Posted Monday Apr 19th, 2010 01:54pm
by Joshua Drake |
Permalink
The great folks of
Linux Fest Nortwest are hosting a PostgreSQL Track again this year. I will be speaking twice. First on what has ended up being a very popular utility,
PITRTools. My second talk will be on Dumb Simple PostgreSQL Performance, which has been very popular with user groups. I look forward to seeing everyone again.
Posted Tuesday Mar 30th, 2010 10:22am
by Joshua Drake |
Permalink
So PgEast is over. You wouldn't know it yet by looking at the website, but it is. We maxed out at ~ 160 people. That is an almost 2x increase over last year at East. It is also a significant increase over West last October. This is an exciting time for this conference series.
The trainings were also successful, I had 19 (of a max 20) show for my Performance and Maintenance class, and I know the other classes had similar attendance.
Due to the success of East, we are going to be moving West to a hotel as well. There was overwhelming support for not returning to a college. Although most were supportive of the reason we used colleges in the past, they were also firm in their belief that further growth of the conferences will require a step up into the professional realm. That means a hotel.
There was also an exciting infrastructure announcement from the .Org sysadmin team. Core team member, Dave Page announced in his talk that .Org will be moving from FreeBSD and jails to Debian Linux and virtualization. This is a long time needed change and I am very glad to be part of the team that will be assisting in this move. I believe that this change will help us entice more people to be part of the sysadmins team and allow for a more diversified and flexible infrastructure.
Copyright © 2000-2010 Command Prompt, Inc. All Rights Reserved. All trademarks property of their respective owners.