Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 3070 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 4.2624. Min Range: -4.2624

Max Act: 4.2624. Min Act: -0.0000

Data Index: 1928725 (The Pile)

Max Activating Token Index: 789

Text #1

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.6901. Min Act: -0.0000

Data Index: 582364 (The Pile)

Max Activating Token Index: 858

Text #2

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.5274. Min Act: -0.0000

Data Index: 1324134 (The Pile)

Max Activating Token Index: 913

Text #3

Max Range: 4.2624. Min Range: -4.2624

Max Act: 2.9928. Min Act: -0.0000

Data Index: 712065 (The Pile)

Max Activating Token Index: 881

Text #4

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.4680. Min Act: -0.0000

Data Index: 876215 (The Pile)

Max Activating Token Index: 748

Text #5

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.1350. Min Act: -0.0000

Data Index: 1275492 (The Pile)

Max Activating Token Index: 654

Text #6

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.2427. Min Act: -0.0000

Data Index: 1247627 (The Pile)

Max Activating Token Index: 972

Text #7

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.2157. Min Act: -0.0000

Data Index: 688569 (The Pile)

Max Activating Token Index: 317

Text #8

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.2172. Min Act: -0.0000

Data Index: 74069 (The Pile)

Max Activating Token Index: 472

Text #9

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.2041. Min Act: -0.0000

Data Index: 581288 (The Pile)

Max Activating Token Index: 578

Text #10

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.2050. Min Act: -0.0000

Data Index: 381921 (The Pile)

Max Activating Token Index: 262

Text #11

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.2000. Min Act: -0.0000

Data Index: 1592749 (The Pile)

Max Activating Token Index: 801

Text #12

Max Range: 4.2624. Min Range: -4.2624

Max Act: 2.9882. Min Act: -0.0000

Data Index: 1525520 (The Pile)

Max Activating Token Index: 544

Text #13

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.2588. Min Act: -0.0000

Data Index: 102823 (The Pile)

Max Activating Token Index: 579

Text #14

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.0584. Min Act: -0.0000

Data Index: 1651295 (The Pile)

Max Activating Token Index: 758

Text #15

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.1990. Min Act: -0.0000

Data Index: 1192608 (The Pile)

Max Activating Token Index: 380

Text #16

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.0832. Min Act: -0.0000

Data Index: 1312560 (The Pile)

Max Activating Token Index: 122

Text #17

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.0724. Min Act: -0.0000

Data Index: 1876433 (The Pile)

Max Activating Token Index: 897

Text #18

Max Range: 4.2624. Min Range: -4.2624

Max Act: 3.0812. Min Act: -0.0000

Data Index: 920621 (The Pile)

Max Activating Token Index: 346

Text #19

Max Range: 4.2624. Min Range: -4.2624

Max Act: 2.9158. Min Act: -0.0000

Data Index: 953930 (The Pile)

Max Activating Token Index: 380