Week 11 – GSoC

So Final Evaluations are due in a few days. I’ve been trying to add a few things to Hydrus to make it more production ready 😛

One of the things that were changed is the actual server that was being run on Hydrus. Hydrus was using the Werkzeug development servers to run the API, we have now switched over to py-greenlet based servers in Gevent. The Werkzeug servers were not able to handle the load of the simulation and would randomly freeze and stop responding. The new servers are run on individual threads and are much more responsive.

I also added authentication features to Hydrus. Although Hydra as a vocabulary doesn’t have native support for authentication, it would be possible to implement this using a custom API Doc. But it would be an overhead for users to add these classes and operations to their own API Docs.

One of the more common ways of authenticating REST APIs is by using the Authorization header in the HTTP request. This was useful, but we needed a way to be able to define the authentication mechanism. One of the things I found for this is the Basic Authentication Scheme defined in this RFC https://tools.ietf.org/html/rfc7617

I implement a SHA224 hash mechanism, where the user will hash his <password>, concatenate it with the <username>, convert the resulting string into base64 codec, and add it to the authorization header, Hydrus will automatically decode these things and authorize the user/request if the credentials are correct. For now, each request needs to be sent along with the Authorization header.

I added an easy way for users to be added during the server setup in the main script and added examples of how it has to be done.

Need to add all the new changes to the Wiki and the documentation now. 🙂

 

Advertisements

Week 10 – GSoC

So much of this week has gone in implementing the things for the simulation, but we’ve finally done it.

There were some problems with Hydrus along the way, but that was resolved. I revamped the CRUD operations also. There were some issues about the initial implementation that was not a good design pattern. We were returning the HTTP response directly from the CRUD operation, which needed to be separated and be restricted to the Views part of the server.

I implemented custom exceptions for each of the bad responses and raised them every time a wrong operation was performed. These exceptions were then caught in the application and an appropriate HTTP response was generated for the same. There was also an issue with the ID not being returned properly for an object. This caused Hydra-py to not be able to use the server endpoints properly.

The controller and drone loops are running and the interactions are occurring as required. Anomaly detection is working too and drones are able to confirm/contradict the Anomalies.

Akshay will now integrate this into the GUI and do the next parts.

Cheers 🙂

Week 9 – GSoC

So this month, we are trying to complete the simulation repos and get everything running.

Akshay has made a nice GUI design for the simulation and now we need to add functionality to the drone and central controller servers by using time loops for each of them.

Lorenzo has proposed a mechanism by which a drone will detect anomalies in the temperature in the simulation and another drone will need to confirm the simulation. We need to figure out a way that this can be done using REST operations alone.

Here is the method I have in mind:

  • Both the server and the drone will be running time loops where they check certain things and take certain actions based on the data in their endpoints.
  • A new endpoint called “Anomaly” is added to both the drone and the central controller, along with a “Collection of Anomaly” at the controller. Anomaly has “Location”, “Temperature” and “DroneID” as its attributes.
  • When a drone detects an anomaly, it pushes the Anomaly to the controller collection. For the simulation, detection of an anomaly is random(1/3 probability of an anomaly in a 15-second loop).
  • In the central controller loop, the controller checks the Collection for any new Anomaly. If an Anomaly is found in the Collection, the server will assign the nearest drone to the location of the Anomaly. It makes sure that the assigned drone is not the same as the drone that detected the Anomaly. Assigning is done by pushing a copy of the Anomaly to the drone endpoint.
  • In every time loop of the drone, the drone checks it’s Anomaly endpoint for any assigned Anomaly. If there is an assigned Anomaly, drone changes its state to “Confirming” and moves towards the Anomaly to confirm/contradict its reading.
  • When a drone is in the “Confirming” state, it does not detect any more Anomaly. Once the drone reaches the Location of the Anomaly, the drone will randomly confirm/contradict it for the purpose of the simulation.

I am going to get these things implemented during the first week and then we will see what needs to be changed in Hydrus as well as in this design to make it work.

That’s it for now 🙂

Week 8 – GSoC

Another revamp for Hydrus! 😎

So another major part of Hydrus got scrapped and reconstructed. The DB connectors we were using(SQLAlchemy) were defined in the application and when the app runs, the connection is established through the connector defined in Hydrus. There wasn’t much we could do about that and no actual way to change it during runtime, this was a problem mainly for writing tests. When testing the app, any data that was added, was pushed directly to the main database, because of the connector.

Earlier,  we had isolated the data operations from the actual server by creating abstract methods in crud.py. The application would just plug in the connector to these operations and do the work. We could easily test the app(without doing any operations) and the crud.py operations using another connector.

Problem? The improved design made the operations and the API Doc largely dependant on each other. We had to check if the documentation allowed endpoints/operation and also the type of data that was being sent in the request. There was no way of checking how these things worked together.

Solution? App contexts 🙂
I had too look for solutions for this problem and came accross something known as an app context in flask. This was basically an object called flask.g that allowed you to store stuff in the app objects itself and use them based on the context in which the app was used.

Simple example: You use one connector when the actual app is running(main database), and you can swap out the connector when testing for another one(test database). I know flask has built in support for most of this, but since we never actually used the ORM from flask, it was pretty cool to implement this functionality on my own.

So, new commits and 10 new tests added :). [commit]

The good news is that all of them passed without any problems.

Another cool thing: Hydrus now actually reads the API Documentation and creates objects based on the specification mentioned in the doc and then runs tests using those objects. So no matter what API Doc is there on the app, the tests will work for them all 🙂

 

 

Week 7 – GSoC(pt.2)

And Hydrus just had a total makeover 😎

Here’s the commit.

Everything uses the API Doc writer. Contexts are generated when classes are added in the begginning. EntryPoint is created and stored in the HydraDoc object as a EntryPoint object. All classes are stored as HydraClass objects, if specified a collection for each class is created as HydraCollection. All operations are stored as HydraClassOp in the classes and EntrypointOp in the entrypoint. Supported properties for classes are stored as HydraClassProp objects. All in all, I feel good adding this to Hydrus, way more efficient than what we had before.

Also made a few changes to Hydrus design, will improve things and add tests soon for Hydrus.

Meanwhile, Akshay is working on the simulation part, when Hydrus is ready, it should only be a matter of plug and play 🙂

Week 7 – GSoC

Finished Doc writer 🙂

Here’s the Commit.

So now, we have this awesome abstraction that can create Hydra API Docs, generate EntryPoints, Contexts and a bunch of other stuff.

Now that this is done, I want to add this to Hydrus. Right now, Hydrus handles the docs part in a very messy way, most of the stuff is generated for every request from something called parsed_classes. Akshay had added this during the first stage to get things working, but it’s high time we have an abstraction for it.

Also changed the API Doc for the simulation some more. All of it is on the writer branch in https://github.com/HTTP-APIs/hydra-flock-demo.

More updates when I add this to Hydrus.

 

Week 6 – GSoC

So we have distributed our work for now. I’m writing the API Doc for the simulation while Akshay works on the simulation implementation. We have a basic design in mind, this is subject to change in the coming days, but will be mostly followed:

design

Basically the endpoints can be independently setup using just an API Documentation and Hydrus.

Once that is done, the controllers will use the endpoints to do the simulation. This will keep the REST API separate for the simulation and will also showcase how we can create an API just by using the Hydra API Doc. Hydrus needs to be worked upon a lot.

I also wrote a couple of gists for the functions that may be needed on the drones for simulation.

Here’s one for the server:

<pre>"""Pseudo code for drone control server."""

class Server():

  def __init__(self, drones):
    now = time()
    self.drones = dict()
    for drone in drones:
      self.drones[drone.id] = drone
    self.log = [LogEntry("Created server at %s" % (now), now)]
    self.data = list()

  def retrieve_drone_list(self):
    now = time()
    event = LogEntry("All Drones requested at %s" % (now), now)
    self.log.append(event)
    return self.drones

  def retrieve_position(self, id_):
    now = time()
    event = LogEntry("Drone %s position requested at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].position

  def retrieve_speed(self, id_):
    now = time()
    event = LogEntry("Drone %s speed requested at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].speed

  def retrieve_battery(self, id_):
    now = time()
    event = LogEntry("Drone %s battery status requested at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].battery

  def retrieve_drone_current_action(self, id_):
    now = time()
    event = LogEntry("Drone %s status requested at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].status

  def change_position(self, id_, position):
    now = time()
    event = LogEntry("Drone %s position changed at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].update_drone_position(position)

  def change_speed(self, id_, speed):
    now = time()
    event = LogEntry("Drone %s speed changed at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].update_drone_speed(speed)

  def change_status(self, id_, status):
    now = time()
    event = LogEntry("Drone %s status changed at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].update_drone_status(status)

  def log_data(self, id_, data):
    now = time()
    event = LogEntry("Drone %s status changed at %s" % (id_,now), now)
    self.data.append(data)

  def coordinate(self, drones):
    now = time()
    event = LogEntry("Drone coordinated at %s" % (now), now)
    for drone in drones:
      # Run some algorithm
      drone.position = new_position

  def recharge_drone(self, id_, charge_time, rate, power_source):
    elapsed = time()
    now = time()
    self.drones[id_].recharge(power_source, rate)
    while now - elapsed < charge_time:
      now = time()
    self.drones[id_].discharge()</pre>

And one for the drone:

<pre>"""Pseudo code for drone control server."""

class Drone():

  def __init__(self, drones):
    now = time()
    self.drones = dict()
    for drone in drones:
      self.drones[drone.id] = drone
    self.log = [LogEntry("Created server at %s" % (now), now)]
    self.data = list()

  def retrieve_drone_list(self):
    now = time()
    event = LogEntry("All Drones requested at %s" % (now), now)
    self.log.append(event)
    return self.drones

  def retrieve_position(self, id_):
    now = time()
    event = LogEntry("Drone %s position requested at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].position

  def retrieve_speed(self, id_):
    now = time()
    event = LogEntry("Drone %s speed requested at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].speed

  def retrieve_battery(self, id_):
    now = time()
    event = LogEntry("Drone %s battery status requested at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].battery

  def retrieve_drone_current_action(self, id_):
    now = time()
    event = LogEntry("Drone %s status requested at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].status

  def change_position(self, id_, position):
    now = time()
    event = LogEntry("Drone %s position changed at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].update_drone_position(position)

  def change_speed(self, id_, speed):
    now = time()
    event = LogEntry("Drone %s speed changed at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].update_drone_speed(speed)

  def change_status(self, id_, status):
    now = time()
    event = LogEntry("Drone %s status changed at %s" % (id_,now), now)
    self.log.append(event)
    return self.drones[id_].update_drone_status(status)

  def log_data(self, id_, data):
    now = time()
    event = LogEntry("Drone %s status changed at %s" % (id_,now), now)
    self.data.append(data)

  def coordinate(self, drones):
    now = time()
    event = LogEntry("Drone coordinated at %s" % (now), now)
    for drone in drones:
      # Run some algorithm
      drone.position = new_position

  def recharge_drone(self, id_, charge_time, rate, power_source):
    elapsed = time()
    now = time()
    self.drones[id_].recharge(power_source, rate)
    while now - elapsed < charge_time:
      now = time()
    self.drones[id_].discharge()

</pre>

I’ve also started writing a doc_writer, for easy creation of a Hydra API Doc, right now it’s just some functions that arrange things into Python dicts, but I’m planning to create classes for the different parts of the documentation so that we can create the Documentation, EntryPoint, Context and other docs on the fly.

Here is the initial version, let’s see where this will go. This commit also contains the API Docs I designed for the central-server and drone.

That’s it for now 🙂