Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4048 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.2286. Min Range: -2.2286

Max Act: 2.2286. Min Act: -0.0000

Data Index: 1358213 (C4 (Web Text))

Max Activating Token Index: 162

Text #1

Max Range: 2.2286. Min Range: -2.2286

Max Act: 2.1489. Min Act: -0.0000

Data Index: 1321620 (C4 (Web Text))

Max Activating Token Index: 303

Text #2

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.9912. Min Act: -0.0000

Data Index: 85383 (C4 (Web Text))

Max Activating Token Index: 10

Text #3

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.9630. Min Act: -0.0000

Data Index: 670797 (C4 (Web Text))

Max Activating Token Index: 377

Text #4

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.8445. Min Act: -0.0000

Data Index: 687915 (C4 (Web Text))

Max Activating Token Index: 374

Text #5

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.9145. Min Act: -0.0000

Data Index: 645048 (C4 (Web Text))

Max Activating Token Index: 420

Text #6

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.7523. Min Act: -0.0000

Data Index: 745614 (C4 (Web Text))

Max Activating Token Index: 310

Text #7

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.8562. Min Act: -0.0000

Data Index: 1171564 (C4 (Web Text))

Max Activating Token Index: 541

Text #8

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.6355. Min Act: -0.0000

Data Index: 934906 (C4 (Web Text))

Max Activating Token Index: 983

Text #9

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.7369. Min Act: -0.0000

Data Index: 1029246 (C4 (Web Text))

Max Activating Token Index: 892

Text #10

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.7599. Min Act: -0.0000

Data Index: 65885 (C4 (Web Text))

Max Activating Token Index: 352

Text #11

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.7078. Min Act: -0.0000

Data Index: 1074940 (C4 (Web Text))

Max Activating Token Index: 463

Text #12

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.5077. Min Act: -0.0000

Data Index: 943014 (C4 (Web Text))

Max Activating Token Index: 330

Text #13

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.6403. Min Act: -0.0000

Data Index: 925433 (C4 (Web Text))

Max Activating Token Index: 889

Text #14

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.5229. Min Act: -0.0000

Data Index: 1074106 (C4 (Web Text))

Max Activating Token Index: 681

Text #15

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.5202. Min Act: -0.0000

Data Index: 213447 (C4 (Web Text))

Max Activating Token Index: 15

Text #16

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.5117. Min Act: -0.0000

Data Index: 645021 (C4 (Web Text))

Max Activating Token Index: 17

Text #17

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.5522. Min Act: -0.0000

Data Index: 53724 (C4 (Web Text))

Max Activating Token Index: 244

Text #18

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.3573. Min Act: -0.0000

Data Index: 588222 (C4 (Web Text))

Max Activating Token Index: 626

Text #19

Max Range: 2.2286. Min Range: -2.2286

Max Act: 1.3392. Min Act: -0.0000

Data Index: 176335 (C4 (Web Text))

Max Activating Token Index: 967