Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2453 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.2466. Min Act: -0.0000

Data Index: 1144731 (C4 (Web Text))

Max Activating Token Index: 288

Text #1

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.2147. Min Act: -0.0000

Data Index: 698940 (C4 (Web Text))

Max Activating Token Index: 266

Text #2

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1991. Min Act: -0.0000

Data Index: 134968 (C4 (Web Text))

Max Activating Token Index: 837

Text #3

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1895. Min Act: -0.0000

Data Index: 785172 (C4 (Web Text))

Max Activating Token Index: 300

Text #4

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.2140. Min Act: -0.0000

Data Index: 813090 (C4 (Web Text))

Max Activating Token Index: 694

Text #5

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.2274. Min Act: -0.0000

Data Index: 1094974 (C4 (Web Text))

Max Activating Token Index: 651

Text #6

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.2056. Min Act: -0.0000

Data Index: 1284348 (C4 (Web Text))

Max Activating Token Index: 118

Text #7

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.2033. Min Act: -0.0000

Data Index: 184363 (C4 (Web Text))

Max Activating Token Index: 315

Text #8

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1869. Min Act: -0.0000

Data Index: 667352 (C4 (Web Text))

Max Activating Token Index: 920

Text #9

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.2036. Min Act: -0.0000

Data Index: 846520 (C4 (Web Text))

Max Activating Token Index: 420

Text #10

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.2019. Min Act: -0.0000

Data Index: 63339 (C4 (Web Text))

Max Activating Token Index: 1017

Text #11

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1884. Min Act: -0.0000

Data Index: 421310 (C4 (Web Text))

Max Activating Token Index: 89

Text #12

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1879. Min Act: -0.0000

Data Index: 1269493 (C4 (Web Text))

Max Activating Token Index: 984

Text #13

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1875. Min Act: -0.0000

Data Index: 1162975 (C4 (Web Text))

Max Activating Token Index: 82

Text #14

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1877. Min Act: -0.0000

Data Index: 1304303 (C4 (Web Text))

Max Activating Token Index: 899

Text #15

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1810. Min Act: -0.0000

Data Index: 138525 (C4 (Web Text))

Max Activating Token Index: 655

Text #16

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1795. Min Act: -0.0000

Data Index: 201603 (C4 (Web Text))

Max Activating Token Index: 688

Text #17

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1858. Min Act: -0.0000

Data Index: 704550 (C4 (Web Text))

Max Activating Token Index: 232

Text #18

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1784. Min Act: -0.0000

Data Index: 1345775 (C4 (Web Text))

Max Activating Token Index: 462

Text #19

Max Range: 0.2466. Min Range: -0.2466

Max Act: 0.1827. Min Act: -0.0000

Data Index: 582476 (C4 (Web Text))

Max Activating Token Index: 549