Model: SoLU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 611 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1511. Min Act: -0.0003

Data Index: 1126572 (C4 (Web Text))

Max Activating Token Index: 738

Text #1

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1477. Min Act: -0.0003

Data Index: 731291 (C4 (Web Text))

Max Activating Token Index: 219

Text #2

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1470. Min Act: -0.0003

Data Index: 1294843 (C4 (Web Text))

Max Activating Token Index: 420

Text #3

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1394. Min Act: -0.0003

Data Index: 47952 (C4 (Web Text))

Max Activating Token Index: 340

Text #4

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1355. Min Act: -0.0003

Data Index: 1035303 (C4 (Web Text))

Max Activating Token Index: 848

Text #5

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1383. Min Act: -0.0003

Data Index: 546867 (C4 (Web Text))

Max Activating Token Index: 743

Text #6

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1354. Min Act: -0.0003

Data Index: 258665 (C4 (Web Text))

Max Activating Token Index: 1022

Text #7

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1390. Min Act: -0.0002

Data Index: 846318 (C4 (Web Text))

Max Activating Token Index: 129

Text #8

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1377. Min Act: -0.0003

Data Index: 1131668 (C4 (Web Text))

Max Activating Token Index: 860

Text #9

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1348. Min Act: -0.0003

Data Index: 553913 (C4 (Web Text))

Max Activating Token Index: 75

Text #10

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1315. Min Act: -0.0003

Data Index: 220359 (C4 (Web Text))

Max Activating Token Index: 422

Text #11

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1344. Min Act: -0.0003

Data Index: 515772 (C4 (Web Text))

Max Activating Token Index: 425

Text #12

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1325. Min Act: -0.0003

Data Index: 600039 (C4 (Web Text))

Max Activating Token Index: 270

Text #13

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1341. Min Act: -0.0003

Data Index: 334923 (C4 (Web Text))

Max Activating Token Index: 715

Text #14

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1316. Min Act: -0.0003

Data Index: 281874 (C4 (Web Text))

Max Activating Token Index: 496

Text #15

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1319. Min Act: -0.0003

Data Index: 1143445 (C4 (Web Text))

Max Activating Token Index: 459

Text #16

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1331. Min Act: -0.0003

Data Index: 711803 (C4 (Web Text))

Max Activating Token Index: 178

Text #17

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1319. Min Act: -0.0003

Data Index: 209110 (C4 (Web Text))

Max Activating Token Index: 775

Text #18

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1280. Min Act: -0.0003

Data Index: 333901 (C4 (Web Text))

Max Activating Token Index: 407

Text #19

Max Range: 0.1511. Min Range: -0.1511

Max Act: 0.1296. Min Act: -0.0003

Data Index: 532756 (C4 (Web Text))

Max Activating Token Index: 125