Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2543 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.7059. Min Act: -0.0002

Data Index: 1637470 (Python Code)

Max Activating Token Index: 327

Text #1

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.4224. Min Act: -0.0002

Data Index: 45467 (C4 (Web Text))

Max Activating Token Index: 326

Text #2

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.5289. Min Act: -0.0002

Data Index: 1458221 (Python Code)

Max Activating Token Index: 529

Text #3

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.3789. Min Act: -0.0002

Data Index: 1381032 (Python Code)

Max Activating Token Index: 927

Text #4

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.0170. Min Act: -0.0002

Data Index: 1441956 (Python Code)

Max Activating Token Index: 278

Text #5

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.1219. Min Act: -0.0002

Data Index: 1415403 (Python Code)

Max Activating Token Index: 267

Text #6

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.3152. Min Act: -0.0002

Data Index: 1563955 (Python Code)

Max Activating Token Index: 200

Text #7

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.1644. Min Act: -0.0002

Data Index: 1532516 (Python Code)

Max Activating Token Index: 9

Text #8

Max Range: 1.7059. Min Range: -1.7059

Max Act: 0.9570. Min Act: -0.0002

Data Index: 1428390 (Python Code)

Max Activating Token Index: 189

Text #9

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.2555. Min Act: -0.0002

Data Index: 1373520 (Python Code)

Max Activating Token Index: 746

Text #10

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.1443. Min Act: -0.0002

Data Index: 1475428 (Python Code)

Max Activating Token Index: 519

Text #11

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.1889. Min Act: -0.0002

Data Index: 1528682 (Python Code)

Max Activating Token Index: 234

Text #12

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.0545. Min Act: -0.0002

Data Index: 1466853 (Python Code)

Max Activating Token Index: 51

Text #13

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.4389. Min Act: -0.0002

Data Index: 1449622 (Python Code)

Max Activating Token Index: 311

Text #14

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.2462. Min Act: -0.0002

Data Index: 1648788 (Python Code)

Max Activating Token Index: 338

Text #15

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.0783. Min Act: -0.0002

Data Index: 1417049 (Python Code)

Max Activating Token Index: 559

Text #16

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.1288. Min Act: -0.0002

Data Index: 1464994 (Python Code)

Max Activating Token Index: 361

Text #17

Max Range: 1.7059. Min Range: -1.7059

Max Act: 1.0883. Min Act: -0.0002

Data Index: 1362100 (Python Code)

Max Activating Token Index: 77

Text #18

Max Range: 1.7059. Min Range: -1.7059

Max Act: 0.9895. Min Act: -0.0002

Data Index: 1499743 (Python Code)

Max Activating Token Index: 954

Text #19

Max Range: 1.7059. Min Range: -1.7059

Max Act: 0.7844. Min Act: -0.0002

Data Index: 1473022 (Python Code)

Max Activating Token Index: 763