Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5893 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 7.3103. Min Range: -7.3103

Max Act: 7.1957. Min Act: -0.0000

Data Index: 270347 (C4 (Web Text))

Max Activating Token Index: 289

Text #1

Max Range: 7.3103. Min Range: -7.3103

Max Act: 6.9897. Min Act: -0.0000

Data Index: 729147 (C4 (Web Text))

Max Activating Token Index: 714

Text #2

Max Range: 7.3103. Min Range: -7.3103

Max Act: 7.3103. Min Act: -0.0000

Data Index: 414556 (C4 (Web Text))

Max Activating Token Index: 765

Text #3

Max Range: 7.3103. Min Range: -7.3103

Max Act: 6.0180. Min Act: -0.0000

Data Index: 128837 (C4 (Web Text))

Max Activating Token Index: 460

Text #4

Max Range: 7.3103. Min Range: -7.3103

Max Act: 5.9493. Min Act: -0.0000

Data Index: 473049 (C4 (Web Text))

Max Activating Token Index: 706

Text #5

Max Range: 7.3103. Min Range: -7.3103

Max Act: 5.8795. Min Act: -0.0000

Data Index: 111486 (C4 (Web Text))

Max Activating Token Index: 1020

Text #6

Max Range: 7.3103. Min Range: -7.3103

Max Act: 5.8927. Min Act: -0.0000

Data Index: 286959 (C4 (Web Text))

Max Activating Token Index: 457

Text #7

Max Range: 7.3103. Min Range: -7.3103

Max Act: 5.7580. Min Act: -0.0000

Data Index: 1180977 (C4 (Web Text))

Max Activating Token Index: 573

Text #8

Max Range: 7.3103. Min Range: -7.3103

Max Act: 5.6449. Min Act: -0.0000

Data Index: 982053 (C4 (Web Text))

Max Activating Token Index: 63

Text #9

Max Range: 7.3103. Min Range: -7.3103

Max Act: 5.1458. Min Act: -0.0000

Data Index: 150016 (C4 (Web Text))

Max Activating Token Index: 119

Text #10

Max Range: 7.3103. Min Range: -7.3103

Max Act: 4.3301. Min Act: -0.0000

Data Index: 376035 (C4 (Web Text))

Max Activating Token Index: 756

Text #11

Max Range: 7.3103. Min Range: -7.3103

Max Act: 4.8533. Min Act: -0.0000

Data Index: 846137 (C4 (Web Text))

Max Activating Token Index: 418

Text #12

Max Range: 7.3103. Min Range: -7.3103

Max Act: 5.0650. Min Act: -0.0000

Data Index: 240419 (C4 (Web Text))

Max Activating Token Index: 596

Text #13

Max Range: 7.3103. Min Range: -7.3103

Max Act: 5.0081. Min Act: -0.0000

Data Index: 2101 (C4 (Web Text))

Max Activating Token Index: 531

Text #14

Max Range: 7.3103. Min Range: -7.3103

Max Act: 4.6213. Min Act: -0.0000

Data Index: 1335325 (C4 (Web Text))

Max Activating Token Index: 184

Text #15

Max Range: 7.3103. Min Range: -7.3103

Max Act: 4.7712. Min Act: -0.0000

Data Index: 347541 (C4 (Web Text))

Max Activating Token Index: 283

Text #16

Max Range: 7.3103. Min Range: -7.3103

Max Act: 4.9582. Min Act: -0.0000

Data Index: 97727 (C4 (Web Text))

Max Activating Token Index: 329

Text #17

Max Range: 7.3103. Min Range: -7.3103

Max Act: 5.0951. Min Act: -0.0000

Data Index: 756103 (C4 (Web Text))

Max Activating Token Index: 817

Text #18

Max Range: 7.3103. Min Range: -7.3103

Max Act: 4.8280. Min Act: -0.0000

Data Index: 570442 (C4 (Web Text))

Max Activating Token Index: 318

Text #19

Max Range: 7.3103. Min Range: -7.3103

Max Act: 4.6251. Min Act: -0.0000

Data Index: 166128 (C4 (Web Text))

Max Activating Token Index: 86