Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2214 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.5065. Min Act: -0.0000

Data Index: 1093745 (C4 (Web Text))

Max Activating Token Index: 94

Text #1

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.4514. Min Act: -0.0000

Data Index: 343765 (C4 (Web Text))

Max Activating Token Index: 460

Text #2

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.4701. Min Act: -0.0000

Data Index: 1030323 (C4 (Web Text))

Max Activating Token Index: 172

Text #3

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.4314. Min Act: -0.0000

Data Index: 1300079 (C4 (Web Text))

Max Activating Token Index: 34

Text #4

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.4157. Min Act: -0.0000

Data Index: 226403 (C4 (Web Text))

Max Activating Token Index: 181

Text #5

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3860. Min Act: -0.0000

Data Index: 917010 (C4 (Web Text))

Max Activating Token Index: 819

Text #6

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3959. Min Act: -0.0000

Data Index: 1103762 (C4 (Web Text))

Max Activating Token Index: 169

Text #7

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3803. Min Act: -0.0000

Data Index: 925739 (C4 (Web Text))

Max Activating Token Index: 973

Text #8

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3644. Min Act: -0.0000

Data Index: 1248623 (C4 (Web Text))

Max Activating Token Index: 1005

Text #9

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3505. Min Act: -0.0000

Data Index: 1001644 (C4 (Web Text))

Max Activating Token Index: 397

Text #10

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3601. Min Act: -0.0000

Data Index: 702651 (C4 (Web Text))

Max Activating Token Index: 162

Text #11

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3826. Min Act: -0.0000

Data Index: 273198 (C4 (Web Text))

Max Activating Token Index: 571

Text #12

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3697. Min Act: -0.0000

Data Index: 1103404 (C4 (Web Text))

Max Activating Token Index: 447

Text #13

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3928. Min Act: -0.0000

Data Index: 153333 (C4 (Web Text))

Max Activating Token Index: 208

Text #14

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3738. Min Act: 0.0000

Data Index: 216000 (C4 (Web Text))

Max Activating Token Index: 76

Text #15

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3672. Min Act: -0.0000

Data Index: 969772 (C4 (Web Text))

Max Activating Token Index: 799

Text #16

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3158. Min Act: -0.0000

Data Index: 75437 (C4 (Web Text))

Max Activating Token Index: 665

Text #17

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3410. Min Act: -0.0000

Data Index: 385234 (C4 (Web Text))

Max Activating Token Index: 710

Text #18

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3202. Min Act: -0.0000

Data Index: 745262 (C4 (Web Text))

Max Activating Token Index: 985

Text #19

Max Range: 0.5065. Min Range: -0.5065

Max Act: 0.3321. Min Act: -0.0000

Data Index: 764521 (C4 (Web Text))

Max Activating Token Index: 323