CREU Research: Privacy in Social Networking

2010-05-29T23:53:00.001-07:00

The research project has officially come to a close. I would like to thank my advisor, Dr. Melody Moh for providing guidance and direction when needed. I would also like to thank the various communities I have been able to bounce ideas off of about the project, and this includes not only fellow students and colleagues, but also various open source communities as well. I have learned a great deal about the research process, and look forward to a possible session of CREU again in the Fall.

The final report summarizes the project, and can be found on my web site at http://www.kizzobot.org, along with other supporting files.

2010-05-15T19:39:00.000-07:00

A rough final draft of the report has been completed, and can be found here. I'll add my final touches in the coming week during finals, and the final version should be done by next weekend.

2010-05-09T12:14:00.001-07:00

The time has come now to write the article. I have already sectioned off the report into discrete chunks that will be filled in later with information. I have begun filling in information and will continue to do so in phases. Lots of information will be added and a lot will be modified to enhance clarity and cohesion.

2010-04-24T12:35:00.000-07:00

I have packaged up the CMake files that I have created for my work with the GNUnet framework, and have placed them on my website for download: http://www.kizzobot.org/ . These CMake files are for 0.8.1b version of the software.

To demonstrate how to make use of the CMake files, follow these instructions. First download the files you'll need: the GNUnet source distribution and the CMake project files. On Ubuntu, you can type "wget http://ftp.gnu.org/pub/gnu/gnunet/GNUnet-0.8.1b.tar.gz" to get the GNUnet code, and "wget http://kizzobot.org/GNUnet-0.8.1b-cmake.tar.bz2" to get the CMake files.

Then, extract the GNUnet source code with "tar xf GNUnet-0.8.1b.tar.gz" . Then extract the GNUnet CMake files to that directory you just extracted from the GNUnet tarball: "tar xf GNUnet-0.8.1b-cmake.tar.bz2 -C GNUnet-0.8.1b" .

Then you should be able to build GNUnet via CMake. Change to the GNUnet source directory and make a new directory where the generated build files from CMake will go: "cd GNUnet-0.8.1b; mkdir build" . Now, change into the build directory that you just created ("cd build") and run cmake to configure. When running cmake, you will need to specify the location of certain things to CMake so that it can proceed. There are only 2 that you must set yourself (GN_USER_HOME_DIR and GN_DAEMON_CONFIG_DIR). Some recommended values are as follows:

GN_USER_HOME_DIR: "/home/kizzo/GNUnet-work/userHome"
GN_DAEMON_CONFIG_DIR: "/home/kizzo/GNUnet-work"
GNUnet_LIBRARY_INSTALL_PATH: "lib/GNUnet"
GNUnet_BINARY_INSTALL_PATH: "bin"
GNUnet_INCLUDE_INSTALL_PATH: "include/GNUnet"
CMAKE_INSTALL_PREFIX: "/home/kizzo/GNUnet-work/install"
CMAKE_INSTALL_RPATH: "/home/kizzo/GNUnet-work/install/lib/GNUnet"

To set these options when running cmake from the build directory, one might use: "cmake .. -DCMAKE_INSTALL_RPATH=/home/kizzo/dsn/install/lib/GNUnet -DCMAKE_INSTALL_PREFIX=/home/kizzo/dsn/install -DGN_USER_HOME_DIR=/home/kizzo/dsn/userHome -DGN_DAEMON_CONFIG_DIR=/home/kizzo/dsn" . Options can be specified on the command line (as shown) or by editing the main CMakeLists.txt file by hand.

After executing this successfully, type "make" to build the libraries and binaries, and then run "make install" to install them to a directories specified by CMAKE_INSTALL_PREFIX .

2010-04-18T11:54:00.000-07:00

I have thought about an end of semester timeline to formally close the project. It looks like there about 4 more weeks before the CREU official end date, and so I thought I should divide things in that way.

This week, from April 18 to April 24, I thought I should package up the CMake build files for easy use later. While my project was unsuccessful in providing a usable, secure, and privacy-preserving alternative, I have managed to contribute a small something that can be useful to someone.

Next week, from April 25 to May 1, I will plan out the project demo, which will showcase what work I have done for the project, and instructions for another developer to lead off on. I am not quite sure yet what I will demo, but this will be fleshed out soon enough - a lot of it will be gathered from journal logs.

The last two weeks, lasting from May 2 to May 15, will be used to write the academic report/article. I'm giving a lot of time to this because I would like for it to be complete.

2010-04-15T20:55:00.001-07:00

I have begun writing the report for the project, summarizing my methods, results, and conclusions. It will be the first academic article I have written.

2010-04-13T22:21:00.001-07:00

I've been spending more time getting the pseudonym list into an array of some sort, and then displaying that in the GUI, with success. It took a lot of weird guess and check work to get the gtk callbacks working, as I had to pass a bunch of widgets around, and I haven't done that much before, but I got used to it eventually.

The semester is coming to an end soon, and it may be likely that I won't be able to continue this project afterwards during the summer, so my advisor said I should probably start thinking about how to wrap things up. I will write an academic article detailing different aspects of the research (results, conclusion, etc.) for publication (my first, I think). I will also have to come up with a demo somehow, but I don't know how to make a good demo out of something that is not complete and does not work as intended, at all. I will think of something though. I was thinking also of packaging up the CMake project files for the GNUnet folks, in case they want them. More on that to come.

2010-04-04T16:28:00.000-07:00

I managed to find out a place to a list of pseudonyms from: the GNUNET_pseudonym_list_all() function. I took a peak at how this function works and it took some time to get used to - trying to remember how passing pointers around work. I'm trying to use this function to get a list of pseudonyms from the framework, so that I can have them displayed in a TreeView widget under the sonet tab.

2010-03-30T17:15:00.001-07:00

Some trouble that I have been running into lately has been worked on. I found out that I had to use Glade to manually change the response ID of the Ok and Cancel button widgets of the New Identity dialog box to GTK_RESPONSE_OK (which equals -5) and GTK_RESPONSE_CANCEL (which equals -6). Before changing these values, the values were both 0, which doesn't correspond to any message that GTK understands, and so I never could wrap my head around that when trying to figure out what a user pressed when prompted by the dialog.

Not all is done however - I still have to get the text out of the newIdentityEntry widget, but that is not going so well so far. More on that tomorrow.

Oh yes - I almost forgot - I haven't been keeping up with my bzr branch these past few weeks, choosing instead to just simply work on the problem locally. I don't know why really, other than just believing, at some point, that maintaining the branch was a waste of time, and so I should stop temporarily. It's not too big of a deal though. The branch has been updated.

2010-03-29T20:11:00.001-07:00

Yesterday ended with me trying to figure out how to get a string value from the user via a dialog box. I want to do this because when the user presses New, the program will need to know the name of the identity that the user wishes to create. Accomplishing this requires writing a bunch of callback functions. I believe that I have to write a callback function that gets called when the Ok button on that dialog is pressed. Then, inside that function, we somehowe retrieve the text in the text entry field of the dialog, and this will be the name that the user has given. With this, we can then load up the main TreeView widegt and then add the new name to the data displayed in it. More on how that goes tomorrow.

2010-03-28T20:10:00.000-07:00

Today, I followed from yesterday and added more widgets to the GUI. There is now a "Social Networking" tab, under which I will place my additional work. There will be a TreeView area to display a list of current identities; a "New" button that, when clicked on, will show a dialog requesting information for a new identity; a "Delete" button to delete a selected identity; an area to the right of the 3 previously mentioned things, used to allow one to edit meta data about a chosen identity; and a bottom area where an identity's wall is displayed and edited. These components will allow one to create and edit identities and their walls.

I spent today learning how to do GUI programming, handling signals and bringing up dialogs at the right times, and so forth. At present time, the Social Networking tab correctly shows and displays the New and Delete buttons, as well as an area below these buttons for the TreeView list of identities. The Delete button does not do anything when clicked, as I have not written a handler for it yet (event-based programming). I have done as such for the New button however, and when clicked, it displays a dialog for more information. I still have to find out how to retrieve the value entered into the dialog, and then use it to create a new identity that will show up in the identities list after the dialog closes.

2010-03-27T23:01:00.000-07:00

I can comfortably use Glade now, both the graphical tool used to visually construct your user interface, and also the libglade library used to extract widget information out of the saved Glade XML file. My task now is to write code that will show my sonet tab when the gnunet-gtk program starts up (that will be at gnunet-gtk/src/plugins/sonet/sonet.c). Inside the init_sonet function there, I will write code to both display the sonet tab, and to display a list of pseudonyms. I have used a debugger on the gnunet-pseudonym utility program to see how a list of pseudonyms can be retrieved, so that I can then learn how to get a list of pseudonyms myself and populate a TreeView GUI widget with them for display. If this is completed, it will be possible to start up gnunet-gtk, click on the "sonet" tab, and see a list of pseudonyms (identities) available. After that, it is planned that I will then add widgets to let one edit these identities.

GUI Programming With Glade

2010-03-14T13:51:00.000-07:00

I find myself looking at the GTK+ reference manual a lot these days in order to make sense of what's going on in the code - in this phase anyway. Glade has been weird to get used to - I've been learning how to use Glade to modify the gnunet-gtk.glade file that is later read from by the plugin code. In the plugin code you're supposed to use the Glade library to extract widget information regarding the gnunet-gtk.glade file, and then populate that widget with data (if I got it right). I am finding it difficult to get my Social Networking tab to show up in the GUI; this could be due to either misuse of the Glade interface designer (saving a weird *.glade file) or bad plugin code (not loading or populating the glade file widgets correctly). I should be able to figure out which is which by next time.

Broken Computer

2010-03-09T23:15:00.000-08:00

My computer's power cable died somehow, so I started running into a bunch of problems related to access to documents that I needed throughout the week for school - very annoying to have this happen in the middle of the semester. Things are back to normal now though (new cord), but moving files from computer to computer just to print something took a lot of time from working. More to come this weekend.

Oh yeah, and if anyone has been looking at my bzr repository for my work so far and noticed that I haven't updated in a while, it is because the GNUnet folks released another new version, and I have just been working on that without pushing my code yet. I'll do that tomorrow as well.

Interface Modifications

2010-02-28T21:00:00.000-08:00

In this phase I am trying to modify the gnunet-gtk GUI so that my new application is displayed. The software uses Glade for its interface design, and I have never used the tool before although I found it to be relatively friendly to use. I have added a tab called "Social Networking" under which all of my other widgets will go.

Glade does not do everything for you though, and so I still have to write GUI callback code in the form of a gnunet-gtk module. I don't have a lot of GUI experience but it's good to get back in touch with it for the project. At this point, I am learning about how to use GTK tree views to display a list of pseudonyms to operate on. I'll post a screenshot of what things look like next time.

Top Down

2010-02-20T14:27:00.000-08:00

2 users should be able to fire up their gnunet-gtk clients and manage identities through them. A user should be able to create and manage many identities (pseudonyms). Each identity will have meta data that the user will be able to edit (first name, last name, interests, etc.). Along with each identity will be a virtual wall, and friends of this identity should be able to (submit changes)/make posts) to this wall.

To that end, the GUI should be modified, and have a new tab made just for social networking aspects of GNUnet. I've decided to do this because I feel that it's better to have all of the related work under one tab, as opposed to making many changes in various other parts of the code to accomplish the same thing. The interface will be primitive: this one tab should allow anyone using GNUnet to create/manage identities and post on friends' walls. To that end, I have begun making using glade to make changes to the GUI. I imagine that there will be widget displaying the current set of identities that exist, and a button for creating one and removing one. There might also be a widget that displays updates, notifying the user that someone has posted something on their wall.

What indicates that 2 identities are friends? What is required for that to happen? How exactly will the wall be implemented, regarding asynchronous modifications and in terms of GNUnet's namespaces?

So Far

2010-02-19T19:10:00.001-08:00

I have been learning more about how Glade interacts with how one creates GUIs. I had trouble seeing how my proposed solution/modifications (see previous post) fit into how everything already works in the framework. More tomorrow.

Long Stretch

2010-02-02T11:06:00.000-08:00

I've been really busy with the start of school and haven't found much time to program, but I did take time to think about the project on a high level, and in the long run. I only have the rest of this semester to work on the project and so I started thinking about a plan for the next few months of school.

Fortunately, I'm not completely in the dark about what needs to be done in order to satisfy project requirements. I need to implement identities, friends, wall, chat, and collab. A user of the system can create any number of identities - they are analogous to the one-and-only identities used for the users of today's popular social networking sites. This functionality is already pretty much done by GNUnet's pseudonym. Friends will need to be implemented, probably by cryptographic signing via keys associated with pseudonyms.

A wall application will be really useful, and so I will need to enable one pseudonym to submit modifications to modify another pseudonym's wall. GNUnet's namespaces should be able to help out with this, as a namespace is a set of files signed by the same pseudonym. With this setup, it looks like a pseudonym's wall can be represented as a file under one of the pseudonym's namespaces. When a friend wants to post to your wall, this can be interpreted as a request from the friend to modify the wall under your namespace. It looks like, for each wall there should be an input queue of modification requests from other pseudonyms. The user can then review these requests (if desired) and accept certain requests and deny others.

Implementing chat sounds like it will be harder than implementing a wall. A wall is just data, whereas chat involves a protocol of some sort. I am still not sure about how to go about this just yet, but I imagine I will come across a solution. As such, chat will likely be implemented only after the wall is.

Collab is an application I'm really excited about and hopefully have time for.

Updated GNUnet Version

2010-01-18T19:27:00.000-08:00

I took some time today to do some branch management, due to the latest development version of the GNUnet framework, 0.9.0 (I was previously using 0.8.0). A few things have changed between the versions, but not so much, and I spent the day learning about bzr's svn plugin to help manage my branches.

The GNUnet folks host the main development repository, and so I reasoned that I should have a bzr branch of my own that mirrors it, and that is located at lp:~/kizzobot/+junk/gnunet . I figured I should do the same with gnunet-gtk, and that is at lp:~/kizzobot/+junk/gnunet-gtk . I also made a branch for the port to CMake, which I am currently working on at lp:~/kizzobot/+junk/gnunet-cmake .

Hopefully by next session, I should have the port to CMake completed for GNUnet. I still have to port gnunet-gtk to CMake, but hopefully that won't be too much of a big deal (you never know though).

CMake Progress

2010-01-17T16:48:00.000-08:00

Yes, I have still been trying to get CMake to work, but the good news is that I am making great progress with it, almost to the point of being finished. During this port to CMake I have ran into lots of roadblocks, but I have come to find that each of them was relatively easy to overcome. A CMake developer noticed my problems with CMake on this blog and offered his help. I took said help and continued plowing away with CMakeLists.txt files. I am happy to say that the port to CMake might be 100% complete by tomorrow, once I setup a mirror for the 0.9.0 release of GNUnet (I have so far been unknowingly porting the 0.8.0 version), and finalize the results to make sure it works just like autotools. More about that next time.

I have also been thinking about the chat module - it seems less complicated than working on collab, to my surprise. In order to implement chat, I would have to look at some (or come up with my own) protocols for P2P chat, and then implement one of them. There are about 2 protocols that I have seen so far to do this. On the other hand, collab seems easier to implement than chat, since that is just file manipulation, and I could leverage the fs module to implement that. I will work on chat though anyway, for now.

Done With CMake

2010-01-12T16:42:00.001-08:00

I have attempted to port GNUnet/gnunet-gtk to CMake but kept running into problems. I would eventually solve a problem but only to run into another. I have convinced myself that I should really stop trying to port to CMake, despite how much I would love that. I have learned more about what still needs to be done to implement peer-to-peer chat in GNUnet, and it will be a long road just to implement that alone. I really need to focus my efforts on implementing P2P chat in GNUnet. In doing so, I will gain a good understanding of how the GNUnet framework is supposed to work, and go from there. Tomorrow I will look at a blog post linked to me by one of the developers about a protocol for chat.

A Little CMake

2010-01-11T11:16:00.000-08:00

I spent all of today trying to get CMake to work with GNUnet and gnunet-gtk, and it was a success. The reason why I went back to CMake despite my own advice to just move on, is because I ran into a compilation problem that I could not figure out, and I believe it had something to do with the autotools scripts. I got frustrated trying to figure it all out and thought I should give CMake a try again.

I really do think I should work with CMake instead of autotools in the long run. It takes way less time to both learn and use CMake than the time it takes to just learn autotools. I ran into a roadblock the last time I worked with CMake but I fortunately solved it relatively quickly. It's good to take some time off of something and come back to it after a while. As things stand today, the latest version of GNUnet (0.9.0) is ported to CMake, with gnunet-gtk in the works tomorrow. Hopefully this solves all compilation/linking issues once and for all so I can focus on other things.

collab

2010-01-05T15:55:00.000-08:00

I got a lot done today. I proceeded from yesterday and added a directory called "collab" that will hold the collaborative document editing code that I will create. I had to modify a bunch of autotools files to inform the build system about the new folder. I copied over all of the files in src/applications/template to start, and changed the names of things from "template" to "collab". I can now run the gnunetd binary but am still running into some issues that should be flushed out by next session. More work tomorrow.

Stop Porting to CMake

2010-01-04T18:28:00.000-08:00

I am finding out more lately about what skills/techniques/methods are needed to do research properly and effectively. Shortly after my last update, I have found that I could speed up my progress even further if I stopped working on the port to CMake and just work with what is already there. I have been running into a few hurdles on the way to fully porting GNUnet/gnunet-gtk to CMake, and so I thought more about a way to get past this, and that way was to just not do it anymore - and I realized that that's fine. Porting to CMake is just something that I would like to have happen (I hate autotools), but I also realised that 1.) it's not necessary, and 2.) working with autotools won't actually be much of a pain because I simply won't be working with it much anyway.

So since my last update, I have been able to successfully compile and run gnunet-gtk, and use it to search and download files on the network. I first tried compiling and running my own gnunetd server, but I initially ran into troubles regarding configuration. I then experimented with using Ubuntu's standard install of gnunet ("aptitude install gnunet-server") to make use of its gnunetd instance, and then ran my own gnunet-gtk client to talk to it, successfully.

I will, however, need to compile and run my own gnunetd because I will need to, of course, add my functionality to it. In that regard, I have spent most of today learning about how to write new applications for GNUnet, and so I spent some time looking in the src/applications/template (of GNUnet) and src/plugins/[about|chat] (of gnunet-gtk) to see how applications are written. Tomorrow, I will get a better understanding of how they work, and begin implementing my own tab in gnunet-gtk, and see how that turns out.

---

I am seeing over and over again how research should be done. It's good to use other's methods to achieve your project's goals because you don't want to duplicate work; you want it to be as easy as possible, and so to that end, it is really useful to leverage the work of other people. Research is about bringing new things into this world and I shouldn't spend time working on something that's already done. I'm also trying to update more often too.

---

I am really excited about the progress so far - I can almost see the finish line and I still have a few months to go. I am seeing now that my output for this project should be a few new folders in the src/plugins directory of gnunet-gtk, and a few new folders in the src/applications directory of GNUnet - and I'm done. Completely done. I now have to learn what needs to be done to properly fill those folders though, that implement useful functionality.

Porting to CMake (continued)

2009-12-24T17:51:00.000-08:00

I am still in the process of porting gnunet-gtk from its original autotools build system to a CMake build system. It took a while to figure out exactly how I am to go about this task, but I eventually narrowed it down by further reading. CMake's documentation is pretty good, and I found what my first steps should be: write a FindGNUNET.cmake to allow the build to find the GNUnet util library and header files, and also write a config.h.cmake file to handle platform-specific build parameters, and also modify the main CMakeLists.txt file to recognize the other two. I had a productive day today, learning more about the powers of CMake. After I am done with config.h.cmake, I believe that I will be able to fully build gnunet-gtk. Then, I will find where plugins are located, and begin studying the workings of those.

Porting to CMake

2009-12-22T13:34:00.000-08:00

Moving on with what I concluded from yesterday, I have made two new branches for me to work on - a branch for GNUnet-0.8.0c named "gnunet", and one for gnunet-gtk-0.8.0c named "gnunet-gtk". My work will mainly take place in gnunet-gtk, but I will need to port the node software as well for libgnunetutil.

My task today was to port the sources from autotools to CMake, as dealing with autotools would be too much of a hastle, and so it is worth the time to make the port. After that is done, I will be able to see more clearly how the frontend is built, so I can find out where and how to write my plugins for it. I will likely have to study the sources of gnunet/src/applications/[template|chat] and gnunet-gtk/src/plugins to see how one writes useful things for the framework, and will then begin on writing my own.

gnunet-gtk plugins

2009-12-21T19:47:00.000-08:00

I thought more about the goals of my project from the beginning and contrasted those thoughts with my intentions now. At the beginning I wanted secure, distributed, anonymous social networking to solve/fix a lot of the privacy/security issues associated with popular social networking sites. I feel that those goals/requirements can be satisfied by the use of GNUnet's primary frontend gnunet-gtk. I am realizing that the core functional requirements of social networking can be implemented by gnunet-gtk, or already are. Chat is in the works, and is in a working state; a wall-like object still doesn't exist yet, but shouldn't be too difficult really - it's just a wall; search and file storage are already implemented by plugins.

All in all, it seems like all of my goals with the project can be satisfied by gnunet-gtk, and all that is needed now is improvements to its functionality and interface. This means that I don't even need to write a full-fledged client any more - all I need to do is write plugins for gnunet-gtk to work with. I have also decided not to write Python bindings any more, as this task is just taking up time and may not be that necessary because I don't mind writing C/C++ that much.

However, I have thought of a killer application for the GNUnet network, and that is collaborative DocBook editing. My goal is to enable large numbers of people to make useful contributions to the same document, primarily DocBook XML documents. And this application can be implemented as a plugin for gnunet-gtk as well.

My goals now are to learn how to write plugins for gnunet-gtk, so that I can later write a plugin for DocBook XML editing. DocBook editing is an issue in itself, and so I will have to take some time to look at some open source implementations. dbkEditor looks like a good start.

I now see that the output of my project, to satisfy goals, should be a set of plugins that can be used by gnunet-gtk.

Python Bindings

2009-12-20T19:56:00.000-08:00

I started working on the Python bindings for GNUnet's util library with Boost.Python, but then began to question how I am doing things, so I took some time to stop coding and verify why I'm doing things and how I should be doing them.

I started to wonder if there were other ways of quickly getting on with the project. I thought about different ways to do the language bindings and have decided to not use Boost.Python because even that would take up too much time, as much as I would like to use it. It might not take up THAT much time, but time is more important than what I would like, so I started considering other ways of doing it. I came across Cython which looks good, and I was going to
use it except that I felt that I was still manually writing everything out, which is almost as long as what I was doing with Boost.Python. I want something that will get me Python bindings in a more automatic way. I fortunately came across a project called pybindgen that is working out really well. I still need to incorporate it though, but that's for tomorrow.

Working With GNUnet

2009-12-19T18:41:00.000-08:00

I spent this week figuring out how to implement my project goals via GNUnet's framework. I began writing Python language bindings for GNUnet's gnunetutil library, and porting their code over from autotools to CMake. Both of these tasks are tedious but need to get done. With this step done, I'll end up with a gnunet.so file that can be used by Python code to access GNUnet's facilities.

I explained the idea to a friend of mine, and that prompted me to think about the long term future of Mav as an open source project. I do not think that it will at all be a problem if like-minding friends help out with this project, and it will get done so much quicker.

Even More Help From FOSS

2009-12-11T18:01:00.000-08:00

I have so far found a new piece of software that is going to save me a ton of work, and it is GNUnet, a framework for secure peer-to-peer networking. I previously planned to use the entangled DHT library to help me code the node component, but that still left me to worry about security issues and routing. With GNUnet, it likes I now won't even have to worry about those things either. From the ground up, GNUnet aims to be a secure, distributed, anonymous, p2p network - and I've come to see that this is exactly what I want out of my project.

It looks like all that I have to do now is write frontend software that communicates to a gnunetd instance (gnunetd pretty much does everything my mavd node was supposed to do). That will be a fun task, as I have now decided not to write a curses-based frontend, and instead write a GUI frontend. A GUI fits naturally with the needs of peer-2-peer social networking, and will have more of an appeal. I must admit, I have been liking Ubuntu+GNOME these days. I'll likely use PyGTK, Glade, and Twisted for the client.

So, now, instead of having to write node software, solve routing, networking, and security problems, all I have to do is write a client program to talk to gnunetd. There is still a lot of work to do though - I still have to figure out how data storage/persistence will work (GNUnet doesn't handle something like that), and learn how to do GUI programming (integrating glade/pygtk/twisted), and learn how to use GNUnet's framework (ECRS and GAP protocols, technicalities, etc.).

From the look of things, I'm making progress, but it is not like it is my fault. GNUnet was always there, and I do remember seeing this project before but didn't look deeply enough to see how good it was. It looks like all of the work that I've done so far - all of the planning, designing, coding, thinking - was a waste of time. I will not be using mavd, as it can now be replaced by gnunetd. I will not be keeping the original mavc. I originally had in mind a curses-based client to use for the user, but I don't feel that kind of interface would be appropriate/flexible enough for social networking applications. A GUI would be far more intuitive for users, and it's not too much different of a thing anyway.

I learned a lot so far actually. I am seeing how so much work is saved by reusing code. At the beginning of this project, I thought I would have to solve routing problems, security problems, networking problems, distributed problems, and all that. Now, it feels like all I'm doing is implementing a plugin that takes care of social networking, and that same plugin is plugged into a framework that takes care of a lot of other work for me. It was evident when I found the twisted/entangled library, and now it's hitting me a second time with GNUnet. Hopefully, secure, decentralized, anonymous, social networking will be available on Ubuntu desktops soon.

Busy Week

2009-11-23T13:27:00.000-08:00

Last week has been really busy for outside reasons, so I did not get much done this week. The node now accepts logins from clients, and one is able to execute the connect, disconnect, login, and logout commands. So I still have to implement creating identities and changing status and whatnot, but I am that much closer to reaching phase 1. After that, I will need to figure out the sharing that will need to happen on the soon-to-be network, and that will involve multiple nodes and clients running.

Protocol, Factory, Deferred

2009-11-13T21:06:00.000-08:00

I spent most of the beginning part of this week getting the client to work right. I spent some working on a more advanced version of the first draft client that made use of twisted, but it took up too much time to learn how to do command line protocols in twisted, so I opted for the simpler standard Python modules to implement this first draft client. This client will be the only client until I write the production client which will likely use curses and twisted.

The latter part of the week I spent getting node code just right, in terms of twisted. Learning how to use this framework is going to take some getting used to, but I can already see the benefits that come along with it - it will definitely pay off in the long run. I am now fairly comfortable with the roles of the Protocol and Factory classes, and am learning how to both use and generate Deferred objects returned by asynchronous functions.

When programming with the twisted framework, you don't call the framework (or anything else) - the framework calls you. The framework runs an event loop that runs continuously, and then calls your methods when network events occur. Each communication channel between the node software and any other entity (be it another node or a client) is represented by an instance of the Protocol class. Each Protocol instance handles the bidirectional communication between two network nodes. However, since each Protocol is instantiated for each connection, and since each connection can come and go at all times, no persistence information is stored in the Protocol objects, and so instead, each Protocol is generated by something called a Factory. A Protocol's Factory (accessed as self.factory) can be used for persistence in the Protocol.

When a Protocol is handling messages, you may need to call a function that may take a long time to complete, like something that requires network or database I/O. To enable concurrency within the framework, there exists an object called a Deferred. A Deferred represents a value that hasn't been computed yet. When a function returns a Deferred, it returns immediately, but that does not mean that the result of that function is available yet. A calling function then attaches callback functions to the Deferred - these functions will be called when the function actually does complete with the result. This allows your program to run in a non-blocking fashion.

So there hasn't been anything too exciting - just getting more familiar with the libraries I just found and how they work. It is still going to take me some time before I reach the first big milestone - a client that can successfully log in to a node, create identities, and change settings. I give it another two weeks to get really acquainted with the libraries and to actually do something useful towards the project with them.

A big leap, thanks to Entangled and Twisted

2009-11-06T21:50:00.000-08:00

Just after I submitted my post for last week, I learned a lot about distributed hash tables and their role/use in peer-to-peer networks. Old generation P2P software relied on centralized methods to share files and find peers. The generation after that tried not to rely on central methods, instead choosing to implement routing via an overlay network. While networks like these don't rely on a central node in any way, routing can be considerably inefficient.

The new generation of P2P networks (as of this writing) make use of distributed hash tables (DHT). DHTs offer many functionalities, including scalability and fault tolerance. I read about the different implementations of DHTs, but the one that caught my eye was that by the name of Kademlia.

I became extremely happy when I found that there was a Python library that was just that - a Python DHT implementation based on Kademlia, called entangled. Entangled makes use of the Twisted network programming framework, and is released under LGPLv3+ (so I will need to make sure that I release my software under a compatible license). Twisted is a network programming framework that I first read about a while ago but never had a use for. I am happy that I am finally getting a chance to work with it.

These two libraries - entangled and Twisted - are going to save me a ton of work. If I didn't have twisted, I would be implementing everything that twisted (pretty much) does for you. My previous node design (see previous post) is now useless thanks to twisted's framework. Now all I do is write handlers for network events - instead of having to code the "watching for" of those events myself.

While twisted takes care of all the low level networking code, entangled takes care of the P2P aspects of things. Entangled provides me with a nice interface to program in a P2P fashion, via DHTs. This project would be greatly hindered if I had to take the time to learn DHTs/Kademlia and implement it myself. Using these two libraries also makes my software more stable, as both libraries are being actively developed.

So now I am taking the time to get reacquainted with programming with twisted, and learning how to use entangled's API to do some useful things. I'm scrapping my old node implementation and reimplementing it with twisted. There is also a lot of work on the client side that still needs to be done. I have reasoned that there will be two client implementations. The first will be a command interpreter intended to meet minimal functionality. The second client will be more polished and intended for initial public release of the software. The second implementation will likely make use of twisted as well, and will have a curses interface similar to irssi's.

Twisted and Entangled
http://www.twistedmatrix.com/
http://entangled.sourceforge.net/

Other links I found this week
http://radar.oreilly.com/2009/09/rss-never-blocks-you-or-goes-d.html
http://www.peerson.net/
http://www.oluyede.org/blog/2008/08/31/twisted-interactive-console/

Node Design

2009-10-30T16:38:00.000-07:00

I am really focused on meeting the first milestone, which is to be able to use mavc to manage a mavd instance with a user account. One should be able to login/logout, create/remove identities, and change status.

I came up with a node design on the bus today and wrote it down on paper. I created a list of terms and wrote a small definition next to each of them: terms such as node, peer (a node that a node knows), client (user frontend), channel (communications channel between 2 nodes or between a node and a client), ChannelManager (a class that will manage a set of channels), and MessageHandler (a class that will process messages).

The node looks like it will consist of 4 parts: a ChannelHandler, a MessageHandler, an input queue, and an output queue. A ChannelHandler is responsible for 1.) populating the input queue with messages received by channels, and 2.) removing messages from the output queue and sending them out on the proper channel. A MessageHandler is responsible for 1.) removing messages from the input queue (for processing), and 2.) populating the output queue with messages.

I figure that a node only communicates with clients and other nodes. Each connection will be called a channel. The ChannelHandler receives messages from channels and place them on the input queue for processing. The MessageHandler removes messages from the input queue, processes them, and puts generated messages on the output queue. The ChannelHandler then removes messages from the output queue and sends them to their intended channel(s). I believe that this architecture will work.

So far, I am still implementing the ChannelHandler, and will work on the MessageHandler when the other 3 components (ChannelHandler, input queue, output queue) have been figured out. Once this architecture is fully implemented, I can then be able to fully implement commands (in the MessageHandler), and be that much closer towards fulfillment of phase 1.

---

I have been thinking a lot about distributed hash tables recently. I don't need to worry about networking/peer-to-peer -related topics/issues just yet - that is for phase 2 - but it is still fun to think about. I understand that there are many ways to implement DHTs and many are available. I came across a kind of DHT called Kademlia that looks really good. There's a Python implementation called SharkyPy that I will likely have to check out when the time comes.

Progress Report

2009-10-17T12:00:00.000-07:00

I so far have a good grasp of what the system is supposed to do (analysis) and how it is supposed to do it (design), but have not fully worked out the details just yet. I have realized the solution as a network of nodes that host identities and their belongings, and also routes traffic. Users run their own node and access it via a client frontend. The next big milestone will be reached when I'm able to run a node and control it with a client; I should be able to use it to change node settings, create/remove identities, and upload content for sharing. The next big milestone after that will be when multiple nodes are able to communicate with each other to share information. I am close to the first milestone though - last night I successfully got the node software (mavd) listening for client connections (mavc). Pretty soon I should be able to use mavc to login to mavd and begin creating identities and uploading sharable content.

The protocol will be a bit tricky - this will be one of the hardest parts of the project, is coming up with this protocol that the system will use. Designing and writing software that implements protocols isn't really hard when compared to the task of designing the protocol itself. I've never really written a protocol before, but I should just have to look at a few protocol specifications to point me in the right direction, like the BitTorrent protocol, Distributed Social Networking Protocol (DSNP), Internet Relay Chat (IRC) protocol, and others.

First Code

2009-10-09T13:18:00.000-07:00

I just started coding first draft implementations of the node and client. The client will be a command interpreter. A more complete client would be a GUI application, so that one could visually manage their identities, and it would also allow visual manipulation of items that are difficult/awkward in a command line only interface (like managing photos and videos). The node will eventually turn into the real production-quality implementation. The goal of these first draft implementations will be to allow one to manage a node. Specifically, this means that the user will be able to use the client program to: create/remove identities, login under identities, change server settings, and edit identities.

I'm really glad that I decided to think about this project in a language-independent way. I have decided that the first full implementation will be written in Python, but I want to make sure that I don't use many (if any) language-specific features - I want to theoretically be able to re implement the whole design in C++ or Java without difficulty, so I am keeping such a desirable outcome in my mind as I am coding the first implementation. This requires staying general, and that is a good thing. Offhand, I know that I will make use of logging, configuration files, doxygen, and threads.

Parts and Pieces

2009-10-02T15:12:00.000-07:00

I believe that my output should be in 3 pieces: node software, client software, and protocol specification. Nodes use the protocol to communicate with other nodes, as well as clients. Clients use the protocol only to communicate with a node (there is no intended client-to-client communication). Nodes route data and information between each other, and clients control nodes. I believe that this kind of structure will fulfill requirements.

The protocol should specify what types of messages are valid and what they mean. The node software will have to be really well done, since it will have a lot of responsibilities, such as routing between nodes and communicating with clients. Since the clients need only to speak and understand our protocol to communicate with nodes, client features may vary widely, from a command line only interface to a GUI. I do not want to write a GUI client, as that will surely take considerable time, and will instead develop a basic text-based command line client (if I even have time to write a good one). A lot of focus should be on the node software though, as it is the backbone component. It will need to be robust and secure.

Next I should think about node designs, so as to figure out exactly how a node does its job.

Analysis and Design

2009-09-25T18:23:00.000-07:00

This past week I thought a lot about the analysis and design aspects of the whole project. I got myself familiar with UML diagrams again, and decided to start practicing making diagrams with the Dia diagram software. I believe that if I am going to have a good chance of success with this project, it will probably be out of my time spent in the analysis and design phase of the project.

I have setup a public bazaar branch at "https://code.launchpad.net/~kizzobot/+junk/mav" which will be used to keep history of the documentation and code. See the README for high level requirements/analysis info. Next week I should have a preliminary set of requirements.

FOAF and the Semantic Web

2009-09-17T09:19:00.000-07:00

Wow! This past week has been really interesting for the project, as I have discovered a few new things of which I have had no experience before. I told a friend of mine about this project that I'm working on, and he suggested something called Friend of a Friend (FOAF). I looked it up later and found that it was something that will play a key role in the software output of the project.

FOAF is a machine-readable ontology intended for use in social networking software. It is used to describe people, their activities, calendars, their documents, and, most importantly, their relationships to other people. It all has to do with a new evolutionary enhancement to the current way that the world-wide web works - a new kind of web called semantic web.

I have heard about semantic web before and was not too interested because I normally don't have anything to do with the web in terms my career/skills in computers. But FOAF is based on prospects of the semantic web. I did some reading about this new semantic web idea and its concepts (RDF, OWL, SPARQL, etc.) and found that it has some appealing uses to me. It basically makes the web (and all of its content) look like a bunch of objects (in object-oriented terminology).

For example, we normally have an HTML document representing a page and a web browser is supposed to parse this document and display its contents in a particular way based on the structure. When you compare this method with the way semantic web works, it sounds like we have been doing it wrong this whole time. In semantic web, the page would be an object, which would be composed of other objects like pictures, links, and paragraphs. The difference is that semantic web makes it clear and precise what the elements are supposed to mean. In comparison to parsing HTML like we do now, elements contained within a paragraph tag may not actually be a paragraph - it could really just be anything. In other words, it's possible to use a paragraph tag for something that is not a paragraph. While this is still possible with semantic web, it is much less likely. Semantics add more "meaning" to what is there.

Using FOAF sounds like a good idea and I believe that it will play a key role in the project. I believe that a lot of my success with this project will be affected by how much I leverage existing standards and implementations. I have found a wealth of related standards that will help me in my goal: Resource Description Framework (RDF), rdflib (programming interface to RDF), FOAF files, Web Ontology Language (OWL), and SPARQL Protocol and RDF Query Language (SPARQL).

I have spent this past week learning, reading, and thinking about FOAF-related concepts and trying to picture a rough draft solution that uses this newly found technology. I still need to do some more work on specifying what the system is to do, and how the system does it. I will start making use-case diagrams and such, in an attempt to visualize a solution involving FOAF and other semantic web standards and protocols.

http://en.wikipedia.org/wiki/FOAF_(software)

Research Plans

2009-09-11T18:33:00.000-07:00

I intend to develop an Internet protocol for use in online social networking, in the form of an RFC. The protocol will allow friends and family to securely share photos, videos, events, files, and messages (email/IM), and other similar functionality. I also intend to develop a draft software implementation of said protocol.

The motivation behind why I want to design and implement a standard and secure protocol for online social networking has a lot to do with the privacy and security implications of the widespread use of social networking sites like Facebook, MySpace, and LiveJournal. One issue of concern is that these sites have the capability to harvest enormous amounts of information about its users. Items of information include: login/logout timestamps, uploaded photos/videos, posts to other users' walls, posts to your wall by other users, events, messages (both email-like and IM), and pretty much anything else that goes through their servers (everything necessarily).

My main point is, I like everything that Facebook does in terms of enabling friends and family to share and communicate information, but what I do not like is having to have all of that information go through their servers. A policy stating that a party will not wrongfully use user information while it is on their servers is not as good as simply not having to have that information be on their servers in the first place.

So I guess my goal with this project is to implement a solution to that problem: do everything that Facebook does but without the central authority. I know that the project sounds Facebook-oriented, and it is, but I believe that that is a good start.

I should probably start by making a list of requirements and restraints. I should start by making a list of things that Facebook does, and then for each one of those things, imagine how the same thing would be accomplished with my software.

Starting Off

2009-09-03T12:44:00.000-07:00

Hello there. My name is David Kilgore, junior undergraduate of Computer Science at San Jose State University, participating as a student in the Collaborative Research Experience for Undergraduates (CREU). I will be making weekly updates regarding my learning experiences, and progress on the design and development of distributed social networking software.

For more information about the program see http://www.cra-w.org/creu .