Turn on the oven to 350 degrees. Put the potatoes in the oven and bake until soft, around 30 to 45 minutes. Remove from the oven and let cool. Cut in half and scoop out filling, leaving about 1/4 an inch of potato in from the skin. Drizzle olive oil on the skin side, place skin side up back in the oven and broil for five minutes. Flip it over, put oil on the flesh side and broil for five minutes.