Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5074 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.8992. Min Act: -0.0000

Data Index: 271652 (C4 (Web Text))

Max Activating Token Index: 218

Text #1

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.5834. Min Act: -0.0000

Data Index: 435126 (C4 (Web Text))

Max Activating Token Index: 307

Text #2

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.6522. Min Act: -0.0000

Data Index: 1072049 (C4 (Web Text))

Max Activating Token Index: 295

Text #3

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.3862. Min Act: -0.0000

Data Index: 1029495 (C4 (Web Text))

Max Activating Token Index: 851

Text #4

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.3932. Min Act: -0.0000

Data Index: 487027 (C4 (Web Text))

Max Activating Token Index: 798

Text #5

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.3788. Min Act: -0.0000

Data Index: 972685 (C4 (Web Text))

Max Activating Token Index: 229

Text #6

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.5186. Min Act: -0.0000

Data Index: 904906 (C4 (Web Text))

Max Activating Token Index: 591

Text #7

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.3331. Min Act: -0.0000

Data Index: 1161839 (C4 (Web Text))

Max Activating Token Index: 850

Text #8

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.2954. Min Act: -0.0000

Data Index: 102061 (C4 (Web Text))

Max Activating Token Index: 826

Text #9

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.2762. Min Act: -0.0000

Data Index: 480320 (C4 (Web Text))

Max Activating Token Index: 253

Text #10

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.2657. Min Act: -0.0000

Data Index: 713260 (C4 (Web Text))

Max Activating Token Index: 559

Text #11

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.2789. Min Act: -0.0000

Data Index: 465306 (C4 (Web Text))

Max Activating Token Index: 409

Text #12

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.3887. Min Act: -0.0000

Data Index: 349567 (C4 (Web Text))

Max Activating Token Index: 838

Text #13

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.2329. Min Act: -0.0000

Data Index: 1189510 (C4 (Web Text))

Max Activating Token Index: 998

Text #14

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.2915. Min Act: -0.0000

Data Index: 704588 (C4 (Web Text))

Max Activating Token Index: 584

Text #15

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.2429. Min Act: -0.0000

Data Index: 607857 (C4 (Web Text))

Max Activating Token Index: 864

Text #16

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.2092. Min Act: -0.0000

Data Index: 18539 (C4 (Web Text))

Max Activating Token Index: 485

Text #17

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.1724. Min Act: -0.0000

Data Index: 566646 (C4 (Web Text))

Max Activating Token Index: 317

Text #18

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.1570. Min Act: -0.0000

Data Index: 1019072 (C4 (Web Text))

Max Activating Token Index: 140

Text #19

Max Range: 1.8992. Min Range: -1.8992

Max Act: 1.1152. Min Act: -0.0000

Data Index: 1167577 (C4 (Web Text))

Max Activating Token Index: 140