Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3369 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.4539. Min Act: -0.0000

Data Index: 267159 (C4 (Web Text))

Max Activating Token Index: 377

Text #1

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.4449. Min Act: -0.0000

Data Index: 373116 (C4 (Web Text))

Max Activating Token Index: 574

Text #2

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.3698. Min Act: -0.0000

Data Index: 134749 (C4 (Web Text))

Max Activating Token Index: 465

Text #3

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.1950. Min Act: -0.0000

Data Index: 224809 (C4 (Web Text))

Max Activating Token Index: 412

Text #4

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.9777. Min Act: -0.0000

Data Index: 990558 (C4 (Web Text))

Max Activating Token Index: 764

Text #5

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.1107. Min Act: -0.0000

Data Index: 1291249 (C4 (Web Text))

Max Activating Token Index: 425

Text #6

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.0838. Min Act: -0.0000

Data Index: 332185 (C4 (Web Text))

Max Activating Token Index: 629

Text #7

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.9502. Min Act: -0.0000

Data Index: 94740 (C4 (Web Text))

Max Activating Token Index: 867

Text #8

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.0430. Min Act: -0.0000

Data Index: 2776 (C4 (Web Text))

Max Activating Token Index: 292

Text #9

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.0113. Min Act: -0.0000

Data Index: 933481 (C4 (Web Text))

Max Activating Token Index: 277

Text #10

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.1243. Min Act: -0.0000

Data Index: 940539 (C4 (Web Text))

Max Activating Token Index: 243

Text #11

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.9188. Min Act: -0.0000

Data Index: 475410 (C4 (Web Text))

Max Activating Token Index: 270

Text #12

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.7643. Min Act: -0.0000

Data Index: 203170 (C4 (Web Text))

Max Activating Token Index: 247

Text #13

Max Range: 2.4539. Min Range: -2.4539

Max Act: 2.0526. Min Act: -0.0000

Data Index: 270346 (C4 (Web Text))

Max Activating Token Index: 769

Text #14

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.9449. Min Act: -0.0000

Data Index: 1149118 (C4 (Web Text))

Max Activating Token Index: 746

Text #15

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.9196. Min Act: -0.0000

Data Index: 260368 (C4 (Web Text))

Max Activating Token Index: 881

Text #16

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.9231. Min Act: -0.0000

Data Index: 846730 (C4 (Web Text))

Max Activating Token Index: 961

Text #17

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.6804. Min Act: -0.0000

Data Index: 450554 (C4 (Web Text))

Max Activating Token Index: 727

Text #18

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.9129. Min Act: -0.0000

Data Index: 51676 (C4 (Web Text))

Max Activating Token Index: 190

Text #19

Max Range: 2.4539. Min Range: -2.4539

Max Act: 1.8925. Min Act: -0.0000

Data Index: 792170 (C4 (Web Text))

Max Activating Token Index: 521